Gelly: Flink Graph API

Gelly is a Graph API for Flink. It contains a set of methods and utilities which aim to simplify the development of graph analysis applications in Flink. In Gelly, graphs can be transformed and modified using high-level functions similar to the ones provided by the batch processing API. Gelly provides methods to create, transform and modify graphs, as well as a library of graph algorithms.

Using Gelly

Gelly is currently part of the libraries Maven project. All relevant classes are located in the org.apache.flink.graph package.

Add the following dependency to your pom.xml to use Gelly.

Java

  1. <dependency>
  2.     <groupId>org.apache.flink</groupId>
  3.     <artifactId>flink-gelly</artifactId>
  4.     <version>1.16.0</version>
  5. </dependency>

Copied to clipboard!

Scala

  1. <dependency>
  2.     <groupId>org.apache.flink</groupId>
  3.     <artifactId>flink-gelly-scala_2.12</artifactId>
  4.     <version>1.16.0</version>
  5. </dependency>

Copied to clipboard!

Note that Gelly is not part of the binary distribution. See linking for instructions on packaging Gelly libraries into Flink user programs.

The remaining sections provide a description of available methods and present several examples of how to use Gelly and how to mix it with the Flink DataSet API.

Running Gelly Examples

The Gelly library jars are provided in the [Flink distribution](http://nightlies.apache.org/flink/flink-docs-release-1.16/docs/libs/gelly/overview/

  1. /flink.apache.org/downloads.html

“Apache Flink: Downloads”) in the opt directory (for versions older than Flink 1.2 these can be manually downloaded from Maven Central). To run the Gelly examples the flink-gelly (for Java) or flink-gelly-scala (for Scala) jar must be copied to Flink’s lib directory.

  1. cp opt/flink-gelly_*.jar lib/
  2. cp opt/flink-gelly-scala_*.jar lib/

Gelly’s examples jar includes drivers for each of the library methods and is provided in the examples directory. After configuring and starting the cluster, list the available algorithm classes:

  1. ./bin/start-cluster.sh
  2. ./bin/flink run examples/gelly/flink-gelly-examples_*.jar

The Gelly drivers can generate graph data or read the edge list from a CSV file (each node in a cluster must have access to the input file). The algorithm description, available inputs and outputs, and configuration are displayed when an algorithm is selected. Print usage for JaccardIndex:

  1. ./bin/flink run examples/gelly/flink-gelly-examples_*.jar --algorithm JaccardIndex

Display graph metrics for a million vertex graph:

  1. ./bin/flink run examples/gelly/flink-gelly-examples_*.jar \
  2.     --algorithm GraphMetrics --order directed \
  3.     --input RMatGraph --type integer --scale 20 --simplify directed \
  4.     --output print

The size of the graph is adjusted by the --scale and --edge_factor parameters. The library generator provides access to additional configuration to adjust the power-law skew and random noise.

Sample social network data is provided by the Stanford Network Analysis Project. The com-lj data set is a good starter size. Run a few algorithms and monitor the job progress in Flink’s Web UI:

  1. wget -O - http://snap.stanford.edu/data/bigdata/communities/com-lj.ungraph.txt.gz | gunzip -c > com-lj.ungraph.txt
  3. ./bin/flink run -q examples/gelly/flink-gelly-examples_*.jar \
  4.     --algorithm GraphMetrics --order undirected \
  5.     --input CSV --type integer --simplify undirected --input_filename com-lj.ungraph.txt --input_field_delimiter $'\t' \
  6.     --output print
  8. ./bin/flink run -q examples/gelly/flink-gelly-examples_*.jar \
  9.     --algorithm ClusteringCoefficient --order undirected \
  10.     --input CSV --type integer --simplify undirected --input_filename com-lj.ungraph.txt --input_field_delimiter $'\t' \
  11.     --output hash
  13. ./bin/flink run -q examples/gelly/flink-gelly-examples_*.jar \
  14.     --algorithm JaccardIndex \
  15.     --input CSV --type integer --simplify undirected --input_filename com-lj.ungraph.txt --input_field_delimiter $'\t' \
  16.     --output hash

Please submit feature requests and report issues on the user mailing list or Flink Jira. We welcome suggestions for new algorithms and features as well as code contributions.