Project Template for Java

Build Tools

Flink projects can be built with different build tools.In order to get started quickly, Flink provides project templates for the following build tools:

  • Maven
  • GradleThese templates help you to set up the project structure and to create the initial build files.

Maven

Requirements

The only requirements are working Maven 3.0.4 (or higher) and Java 8.x installations.

Create Project

Use one of the following commands to create a project:

  1.     $ mvn archetype:generate                               \
  2.       -DarchetypeGroupId=org.apache.flink              \
  3.       -DarchetypeArtifactId=flink-quickstart-java      \
  4.       -DarchetypeVersion=1.9.0
  5.

This allows you to name your newly created project. It will interactively ask you for the groupId, artifactId, and package name. 

  1.     $ curl https://flink.apache.org/q/quickstart.sh | bash -s 1.9.0
  3.

Inspect Project

There will be a new directory in your working directory. If you’ve usedthe curl approach, the directory is called quickstart. Otherwise,it has the name of your artifactId:

  1. $ tree quickstart/
  2. quickstart/
  3. ├── pom.xml
  4. └── src
  5.     └── main
  6.         ├── java
  7.         │   └── org
  8.         │       └── myorg
  9.         │           └── quickstart
  10.         │               ├── BatchJob.java
  11.         │               └── StreamingJob.java
  12.         └── resources
  13.             └── log4j.properties

The sample project is a Maven project, which contains two classes: StreamingJob and BatchJob are the basic skeleton programs for a DataStream and DataSet program.The main method is the entry point of the program, both for in-IDE testing/execution and for proper deployments.

We recommend you import this project into your IDE to develop andtest it. IntelliJ IDEA supports Maven projects out of the box.If you use Eclipse, the m2e pluginallows to import Maven projects.Some Eclipse bundles include that plugin by default, others require youto install it manually.

Please note: The default JVM heapsize for Java may be toosmall for Flink. You have to manually increase it.In Eclipse, choose Run Configurations -> Arguments and write into the VM Arguments box: -Xmx800m.In IntelliJ IDEA recommended way to change JVM options is from the Help | Edit Custom VM Options menu. See this article for details.

Build Project

If you want to build/package your project, go to your project directory andrun the ‘mvn clean package’ command.You will find a JAR file that contains your application, plus connectors and librariesthat you may have added as dependencies to the application: target/<artifact-id>-<version>.jar.

Note: If you use a different class than StreamingJob as the application’s main class / entry point,we recommend you change the mainClass setting in the pom.xml file accordingly. That way, Flinkcan run the application from the JAR file without additionally specifying the main class.

Gradle

Requirements

The only requirements are working Gradle 3.x (or higher) and Java 8.x installations.

Create Project

Use one of the following commands to create a project:

  1. buildscript {
  2.     repositories {
  3.         jcenter() // this applies only to the Gradle 'Shadow' plugin
  4.     }
  5.     dependencies {
  6.         classpath 'com.github.jengelman.gradle.plugins:shadow:2.0.4'
  7.     }
  8. }
  10. plugins {
  11.     id 'java'
  12.     id 'application'
  13.     // shadow plugin to produce fat JARs
  14.     id 'com.github.johnrengelman.shadow' version '2.0.4'
  15. }
  18. // artifact properties
  19. group = 'org.myorg.quickstart'
  20. version = '0.1-SNAPSHOT'
  21. mainClassName = 'org.myorg.quickstart.StreamingJob'
  22. description = """Flink Quickstart Job"""
  24. ext {
  25.     javaVersion = '1.8'
  26.     flinkVersion = '1.9.0'
  27.     scalaBinaryVersion = '2.11'
  28.     slf4jVersion = '1.7.7'
  29.     log4jVersion = '1.2.17'
  30. }
  33. sourceCompatibility = javaVersion
  34. targetCompatibility = javaVersion
  35. tasks.withType(JavaCompile) {
  36.     options.encoding = 'UTF-8'
  37. }
  39. applicationDefaultJvmArgs = ["-Dlog4j.configuration=log4j.properties"]
  41. task wrapper(type: Wrapper) {
  42.     gradleVersion = '3.1'
  43. }
  45. // declare where to find the dependencies of your project
  46. repositories {
  47.     mavenCentral()
  48.     maven { url "https://repository.apache.org/content/repositories/snapshots/" }
  49. }
  51. // NOTE: We cannot use "compileOnly" or "shadow" configurations since then we could not run code
  52. // in the IDE or with "gradle run". We also cannot exclude transitive dependencies from the
  53. // shadowJar yet (see https://github.com/johnrengelman/shadow/issues/159).
  54. // -> Explicitly define the // libraries we want to be included in the "flinkShadowJar" configuration!
  55. configurations {
  56.     flinkShadowJar // dependencies which go into the shadowJar
  58.     // always exclude these (also from transitive dependencies) since they are provided by Flink
  59.     flinkShadowJar.exclude group: 'org.apache.flink', module: 'force-shading'
  60.     flinkShadowJar.exclude group: 'com.google.code.findbugs', module: 'jsr305'
  61.     flinkShadowJar.exclude group: 'org.slf4j'
  62.     flinkShadowJar.exclude group: 'log4j'
  63. }
  65. // declare the dependencies for your production and test code
  66. dependencies {
  67.     // --------------------------------------------------------------
  68.     // Compile-time dependencies that should NOT be part of the
  69.     // shadow jar and are provided in the lib folder of Flink
  70.     // --------------------------------------------------------------
  71.     compile "org.apache.flink:flink-java:${flinkVersion}"
  72.     compile "org.apache.flink:flink-streaming-java_${scalaBinaryVersion}:${flinkVersion}"
  74.     // --------------------------------------------------------------
  75.     // Dependencies that should be part of the shadow jar, e.g.
  76.     // connectors. These must be in the flinkShadowJar configuration!
  77.     // --------------------------------------------------------------
  78.     //flinkShadowJar "org.apache.flink:flink-connector-kafka-0.11_${scalaBinaryVersion}:${flinkVersion}"
  80.     compile "log4j:log4j:${log4jVersion}"
  81.     compile "org.slf4j:slf4j-log4j12:${slf4jVersion}"
  83.     // Add test dependencies here.
  84.     // testCompile "junit:junit:4.12"
  85. }
  87. // make compileOnly dependencies available for tests:
  88. sourceSets {
  89.     main.compileClasspath += configurations.flinkShadowJar
  90.     main.runtimeClasspath += configurations.flinkShadowJar
  92.     test.compileClasspath += configurations.flinkShadowJar
  93.     test.runtimeClasspath += configurations.flinkShadowJar
  95.     javadoc.classpath += configurations.flinkShadowJar
  96. }
  98. run.classpath = sourceSets.main.runtimeClasspath
  100. jar {
  101.     manifest {
  102.         attributes 'Built-By': System.getProperty('user.name'),
  103.                 'Build-Jdk': System.getProperty('java.version')
  104.     }
  105. }
  107. shadowJar {
  108.     configurations = [project.configurations.flinkShadowJar]
  109. }
  110.                 
  1. rootProject.name = 'quickstart'
  2.                 
  1.     bash -c "$(curl https://flink.apache.org/q/gradle-quickstart.sh)" -- 1.9.0 2.11
  2.

This allows you to name your newly created project. It will interactively ask you for the project name, organization (also used for the package name), project version, Scala and Flink version.

Inspect Project

There will be a new directory in your working directory based on theproject name you provided, e.g. for quickstart:

  1. $ tree quickstart/
  2. quickstart/
  3. ├── README
  4. ├── build.gradle
  5. ├── settings.gradle
  6. └── src
  7.     └── main
  8.         ├── java
  9.         │   └── org
  10.         │       └── myorg
  11.         │           └── quickstart
  12.         │               ├── BatchJob.java
  13.         │               └── StreamingJob.java
  14.         └── resources
  15.             └── log4j.properties

The sample project is a Gradle project, which contains two classes: StreamingJob and BatchJob are the basic skeleton programs for a DataStream and DataSet program.The main method is the entry point of the program, both for in-IDE testing/execution and for proper deployments.

We recommend you import this project into your IDE to develop andtest it. IntelliJ IDEA supports Gradle projects after installing the Gradle plugin.Eclipse does so via the Eclipse Buildship plugin(make sure to specify a Gradle version >= 3.0 in the last step of the import wizard; the shadow plugin requires it).You may also use Gradle’s IDE integrationto create project files from Gradle.

Please note: The default JVM heapsize for Java may be toosmall for Flink. You have to manually increase it.In Eclipse, choose Run Configurations -> Arguments and write into the VM Arguments box: -Xmx800m.In IntelliJ IDEA recommended way to change JVM options is from the Help | Edit Custom VM Options menu. See this article for details.

Build Project

If you want to build/package your project, go to your project directory andrun the ‘gradle clean shadowJar’ command.You will find a JAR file that contains your application, plus connectors and librariesthat you may have added as dependencies to the application: build/libs/<project-name>-<version>-all.jar.

Note: If you use a different class than StreamingJob as the application’s main class / entry point,we recommend you change the mainClassName setting in the build.gradle file accordingly. That way, Flinkcan run the application from the JAR file without additionally specifying the main class.

Next Steps

Write your application!

If you are writing a streaming application and you are looking for inspiration what to write,take a look at the Stream Processing Application Tutorial.

If you are writing a batch processing application and you are looking for inspiration what to write,take a look at the Batch Application Examples.

For a complete overview over the APIs, have a look at theDataStream API andDataSet API sections.

Here you can find out how to run an application outside the IDE on a local cluster.

If you have any trouble, ask on ourMailing List.We are happy to provide help.