This page provides instructions on how to enable SSL for the network communication between different Flink components.
SSL can be enabled for all network communication between Flink components. SSL keystores and truststore has to be deployed on each Flink node and configured (conf/flink-conf.yaml) using keys in the security.ssl.* namespace (Please see the configuration page for details). SSL can be selectively enabled/disabled for different transports using the following flags. These flags are only applicable when security.ssl.enabled is set to true.
taskmanager.data.ssl.enabled: SSL flag for data communication between task managers
blob.service.ssl.enabled: SSL flag for blob service client/server communication
akka.ssl.enabled: SSL flag for akka based control connection between the Flink client, jobmanager and taskmanager
jobmanager.web.ssl.enabled: Flag to enable https access to the jobmanager’s web frontend
Deploying Keystores and Truststores
You need to have a Java Keystore generated and copied to each node in the Flink cluster. The common name or subject alternative names in the certificate should match the node’s hostname and IP address. Keystores and truststores can be generated using the keytool utility. All Flink components should have read access to the keystore and truststore files.
Example: Creating self signed CA and keystores for a two-node cluster
Execute the following keytool commands to create a truststore with a self signed CA.
Now create keystores for each node with certificates signed by the above CA. Let node1.company.org and node2.company.org be the hostnames with IPs 192.168.1.1 and 192.168.1.2 respectively
Configure each node in the standalone cluster to pick up the keystore and truststore files present in the local file system.
Example: Two-node cluster
Generate two keystores, one for each node, and copy them to the filesystem on the respective node. Also copy the public key of the CA (which was used to sign the certificates in the keystore) as a Java truststore on both the nodes.
Configure conf/flink-conf.yaml to pick up these files.
Restart the Flink components to enable SSL for all of Flink’s internal communication
Verify by accessing the jobmanager’s UI using https url. The taskmanager’s path in the UI should show akka.ssl.tcp:// as the protocol
The blob server and taskmanager’s data communication can be verified from the log files
The keystores and truststore can be deployed in a YARN setup in multiple ways depending on the cluster setup. Following are two ways to achieve this.
1. Deploy keystores before starting the YARN session
The keystores and truststore should be generated and deployed on all nodes in the YARN setup where Flink components can potentially be executed. The same Flink config file from the Flink YARN client is used for all the Flink components running in the YARN cluster. Therefore we need to ensure the keystore is deployed and accessible using the same filepath in all the YARN nodes.
Now you can start the YARN session from the CLI like you would normally do.
2. Use YARN CLI to deploy the keystores and truststore
We can use the YARN client’s ship files option (-yt) to distribute the keystores and truststore. Since the same keystore will be deployed at all nodes, we need to ensure a single certificate in the keystore can be served for all nodes. This can be done by either using the Subject Alternative Name (SAN) extension in the certificate and setting it to cover all nodes (hostname and ip addresses) in the cluster or by using wildcard subdomain names (if the cluster is setup accordingly).
Supply the following parameters to the keytool command when generating the keystore: -ext SAN=dns:node1.company.org,ip:192.168.1.1,dns:node2.company.org,ip:192.168.1.2
Copy the keystore and the CA’s truststore into a local directory (at the CLI’s working directory), say deploy-keys/
Update the configuration to pick up the files from a relative path
Start the YARN session using the -yt parameter
When deployed using YARN, Flink’s web dashboard is accessible through YARN proxy’s Tracking URL. To ensure that the YARN proxy is able to access Flink’s https url you need to configure YARN proxy to accept Flink’s SSL certificates. Add the custom CA certificate into Java’s default truststore on the YARN Proxy node.