Building Flink from Source

This page covers how to build Flink 1.15.0 from sources.

In order to build Flink you need the source code. Either [download the source of a release](http://nightlies.apache.org/flink/flink-docs-release-1.15/docs/flinkdev/building/

  1. /flink.apache.org/downloads.html

) or clone the git repository.

In addition you need Maven 3 and a JDK (Java Development Kit). Flink requires at least Java 11 to build.

NOTE: Maven 3.3.x can build Flink, but will not properly shade away certain dependencies. Maven 3.2.5 creates the libraries properly.

To clone from git, enter:

  1. git clone https://github.com/apache/flink.git

The simplest way of building Flink is by running:

  1. mvn clean install -DskipTests

This instructs Maven (mvn) to first remove all existing builds (clean) and then create a new Flink binary (install).

To speed up the build you can:

  • skip tests by using ‘ -DskipTests’
  • skip QA plugins and javadoc generation by using the fast Maven profile
  • skip the WebUI compilation by using the skip-webui-build Maven profile

  • use Maven’s parallel build feature, e.g., ‘mvn package -T 1C’ will attempt to build 1 module for each CPU core in parallel.

    Parallel builds may deadlock due to a bug in the maven-shade-plugin. It is recommended to only use it as a 2 steps process, where you first run mvn validate/test-compile/test in parallel, and then run mvn package/verify/install with a single thread.

The build script will be:

  1. mvn clean install -DskipTests -Dfast -Pskip-webui-build -T 1C

The fast and skip-webui-build profiles have a significant impact on the build time, particularly on slower storage devices, due to them reading/writing many small files.

Prerequisites

  1. Building Flink

    If you want to build a PyFlink package that can be used for pip installation, you need to build the Flink project first, as described in Build Flink.

  2. Python version(3.6, 3.7 or 3.8) is required

    1. $ python --version
    2. # the version printed here must be 3.6, 3.7 or 3.8

  3. Build PyFlink with Cython extension support (optional)

    To build PyFlink with Cython extension support, you’ll need a C compiler. It’s a little different on how to install the C compiler on different operating systems:

    • Linux Linux operating systems usually come with GCC pre-installed. Otherwise, you need to install it manually. For example, you can install it with command sudo apt-get install build-essential On Ubuntu or Debian.

    • Mac OS X To install GCC on Mac OS X, you need to download and install “Command Line Tools for Xcode”, which is available in Apple’s developer page.

    You also need to install the dependencies with following command:

    1. $ python -m pip install -r flink-python/dev/dev-requirements.txt

Installation

Then go to the root directory of flink source code and run this command to build the sdist package and wheel package of apache-flink and apache-flink-libraries:

  1. cd flink-python; python setup.py sdist bdist_wheel; cd apache-flink-libraries; python setup.py sdist; cd ..;

The sdist package of apache-flink-libraries will be found under ./flink-python/apache-flink-libraries/dist/. It could be installed as following:

  1. python -m pip install apache-flink-libraries/dist/*.tar.gz

The sdist and wheel packages of apache-flink will be found under ./flink-python/dist/. Either of them could be used for installation, such as:

  1. python -m pip install dist/*.whl

Dependency Shading

Flink shades away some of the libraries it uses, in order to avoid version clashes with user programs that use different versions of these libraries. Among the shaded libraries are Google Guava, Asm, Apache Curator, Apache HTTP Components, Netty, and others.

The dependency shading mechanism was recently changed in Maven and requires users to build Flink slightly differently, depending on their Maven version:

Maven and 3.2.x It is sufficient to call mvn clean install -DskipTests in the root directory of Flink code base.

Maven 3.3.x The build has to be done in two steps: First in the base directory, then in shaded modules, such as the distribution and the filesystems:

  1. # build overall project
  2. mvn clean install -DskipTests
  4. # build shaded modules used in dist again, for example:
  5. cd flink-filesystems/flink-s3-fs-presto/
  6. mvn clean install -DskipTests
  7. # ... and other modules
  9. # build dist again to include shaded modules
  10. cd flink-dist
  11. mvn clean install

Note: To check your Maven version, run mvn --version.

Note: We recommend using the latest Maven 3.2.x version for building production-grade Flink distributions, as this is the version the Flink developers are using for the official releases and testing.

Back to top

Scala Versions

Users that purely use the Java APIs and libraries can ignore this section.

Flink has APIs, libraries, and runtime modules written in Scala. Users of the Scala API and libraries may have to match the Scala version of Flink with the Scala version of their projects (because Scala is not strictly backwards compatible).

Since version 1.7 Flink builds with Scala version 2.11 (default) and 2.12.

To build Flink against Scala 2.12, issue the following command:

  1. mvn clean install -DskipTests -Dscala-2.12

To build against a specific binary Scala version you can use:

  1. mvn clean install -DskipTests -Dscala-2.12 -Dscala.version=<scala version>

Back to top

Encrypted File Systems

If your home directory is encrypted you might encounter a java.io.IOException: File name too long exception. Some encrypted file systems, like encfs used by Ubuntu, do not allow long filenames, which is the cause of this error.

The workaround is to add:

  1. <args>
  2.     <arg>-Xmax-classfile-name</arg>
  3.     <arg>128</arg>
  4. </args>

in the compiler configuration of the pom.xml file of the module causing the error. For example, if the error appears in the flink-yarn module, the above code should be added under the <configuration> tag of scala-maven-plugin. See this issue for more information.

Back to top