Flink has a history server that can be used to query the statistics of completed jobs after the corresponding Flink cluster has been shut down.
Furthermore, it exposes a REST API that accepts HTTP requests and responds with JSON data.
The HistoryServer allows you to query the status and statistics of completed jobs that have been archived by a JobManager.
After you have configured the HistoryServer and JobManager, you start and stop the HistoryServer via its corresponding startup script:
By default, this server binds to
localhost and listens at port
Currently, you can only run it as a standalone process.
The configuration keys
historyserver.archive.fs.refresh-interval need to be adjusted for archiving and displaying archived jobs.
The archiving of completed jobs happens on the JobManager, which uploads the archived job information to a file system directory. You can configure the directory to archive completed jobs in
flink-conf.yaml by setting a directory via
The HistoryServer can be configured to monitor a comma-separated list of directories in via
historyserver.archive.fs.dir. The configured directories are regularly polled for new archives; the polling interval can be configured via
The contained archives are downloaded and cached in the local filesystem. The local directory for this is configured via
Check out the configuration page for a complete list of configuration options.
Below is a list of available requests, with a sample JSON response. All requests are of the sample form
http://hostname:8082/jobs, below we list only the path part of the URLs.
Values in angle brackets are variables, for example
http://hostname:port/jobs/<jobid>/exceptions will have to requested for example as