This documentation is for an out-of-date version of Apache Flink. We recommend you use the latest stable version.
  1. Deployment & Operations
  2. Clusters & Deployment
  3. Docker

Docker Setup

Docker is a popular container runtime. There are official Flink Docker images available on Docker Hub which can be used directly or extended to better integrate into a production environment.

The official Flink Docker repository is hosted on Docker Hub and serves images of Flink version 1.2.1 and later.

Images for each supported combination of Hadoop and Scala are available, and tag aliases are provided for convenience.

For example, the following aliases can be used: (1.2.y indicates the latest release of Flink 1.2)

  • flink:latest → flink:<latest-flink>-hadoop<latest-hadoop>-scala_<latest-scala>
  • flink:1.2 → flink:1.2.y-hadoop27-scala_2.11
  • flink:1.2.1-scala_2.10 → flink:1.2.1-hadoop27-scala_2.10
  • flink:1.2-hadoop26 → flink:1.2.y-hadoop26-scala_2.11

Docker Compose is a convenient way to run a group of Docker containers locally.

An example config file is available on GitHub.

Usage

  • Launch a cluster in the foreground

      docker-compose up

  • Launch a cluster in the background

      docker-compose up -d

  • Scale the cluster up or down to N TaskManagers

      docker-compose scale taskmanager=<N>

When the cluster is running, you can visit the web UI at http://localhost:8081 and submit a job.

To submit a job via the command line, you must copy the JAR to the Jobmanager container and submit the job from there.

For example:

$ JOBMANAGER_CONTAINER=$(docker ps --filter name=jobmanager --format={{.ID}})
$ docker cp path/to/jar "$JOBMANAGER_CONTAINER":/job.jar
$ docker exec -t -i "$JOBMANAGER_CONTAINER" flink run /job.jar

Back to top