Local Cluster

Get a local Flink cluster up and running in a few simple steps.

Setup: Download and Start Flink

Flink runs on Linux and Mac OS X. Note: Windows users can run Flink in Cygwin or WSL. To be able to run Flink, the only requirement is to have a working Java 8 or 11 installation.

You can check the correct installation of Java by issuing the following command:

  1. java -version

If you have Java 8, the output will look something like this:

  1. java version "1.8.0_111"
  2. Java(TM) SE Runtime Environment (build 1.8.0_111-b14)
  3. Java HotSpot(TM) 64-Bit Server VM (build 25.111-b14, mixed mode)
  1. Download a binary from the downloads page. You can pick any Scala variant you like. For certain features you may also have to download one of the pre-bundled Hadoop jars and place them into the /lib directory.
  2. Go to the download directory.
  3. Unpack the downloaded archive.
  1. $ cd ~/Downloads        # Go to download directory
  2. $ tar xzf flink-*.tgz   # Unpack the downloaded archive
  3. $ cd flink-1.11.0

For MacOS X users, Flink can be installed through Homebrew.

  1. $ brew install apache-flink
  2. ...
  3. $ flink --version
  4. Version: 1.2.0, Commit ID: 1c659cf

Start a Local Flink Cluster

  1. $ ./bin/start-cluster.sh  # Start Flink

Check the Dispatcher’s web frontend at http://localhost:8081 and make sure everything is up and running. The web frontend should report a single available TaskManager instance.

Dispatcher: Overview

You can also verify that the system is running by checking the log files in the logs directory:

  1. $ tail log/flink-*-standalonesession-*.log
  2. INFO ... - Rest endpoint listening at localhost:8081
  3. INFO ... - http://localhost:8081 was granted leadership ...
  4. INFO ... - Web frontend listening at http://localhost:8081.
  5. INFO ... - Starting RPC endpoint for StandaloneResourceManager at akka://flink/user/resourcemanager .
  6. INFO ... - Starting RPC endpoint for StandaloneDispatcher at akka://flink/user/dispatcher .
  7. INFO ... - ResourceManager akka.tcp://flink@localhost:6123/user/resourcemanager was granted leadership ...
  8. INFO ... - Starting the SlotManager.
  9. INFO ... - Dispatcher akka.tcp://flink@localhost:6123/user/dispatcher was granted leadership ...
  10. INFO ... - Recovering all persisted jobs.
  11. INFO ... - Registering TaskManager ... at ResourceManager

Windows Cygwin Users

If you are installing Flink from the git repository and you are using the Windows git shell, Cygwin can produce a failure similar to this one:

  1. c:/flink/bin/start-cluster.sh: line 30: $'\r': command not found

This error occurs because git is automatically transforming UNIX line endings to Windows style line endings when running in Windows. The problem is that Cygwin can only deal with UNIX style line endings. The solution is to adjust the Cygwin settings to deal with the correct line endings by following these three steps:

  1. Start a Cygwin shell.

  2. Determine your home directory by entering

  1. cd; pwd
  1. This will return a path under the Cygwin root path.
  1. Using NotePad, WordPad or a different text editor open the file .bash_profile in the home directory and append the following: (If the file does not exist you will have to create it)
  1. export SHELLOPTS
  2. set -o igncr

Save the file and open a new bash shell.

Stop a Local Flink Cluster

To stop Flink when you’re done type:

  1. $ ./bin/stop-cluster.sh

You can terminate the processes via CTRL-C in the spawned shell windows.

Back to top