Checkpoints make state in Flink fault tolerant by allowing state and the corresponding stream positions to be recovered, thereby giving the application the same semantics as a failure-free execution.
See Checkpointing for how to enable and configure checkpoints for your program.
Checkpoints are by default not retained and are only used to resume a job from failures. They are deleted when a program is cancelled. You can, however, configure periodic checkpoints to be retained. Depending on the configuration these retained checkpoints are not automatically cleaned up when the job fails or is canceled. This way, you will have a checkpoint around to resume from if your job fails.
ExternalizedCheckpointCleanup mode configures what happens with checkpoints when you cancel the job:
ExternalizedCheckpointCleanup.RETAIN_ON_CANCELLATION: Retain the checkpoint when the job is cancelled. Note that you have to manually clean up the checkpoint state after cancellation in this case.
ExternalizedCheckpointCleanup.DELETE_ON_CANCELLATION: Delete the checkpoint when the job is cancelled. The checkpoint state will only be available if the job fails.
Similarly to savepoints, a checkpoint consists
of a meta data file and, depending on the state backend, some additional data
files. The meta data file and data files are stored in the directory that is
state.checkpoints.dir in the configuration files,
and also can be specified for per job in the code.
The current checkpoint directory layout (introduced by FLINK-8531) is as follows:
The SHARED directory is for state that is possibly part of multiple checkpoints, TASKOWNED is for state that must never by dropped by the JobManager, and EXCLUSIVE is for state that belongs to one checkpoint only.
Checkpoints have a few differences from savepoints. They
A job may be resumed from a checkpoint just as from a savepoint by using the checkpoint’s meta data file instead (see the savepoint restore guide). Note that if the meta data file is not self-contained, the jobmanager needs to have access to the data files it refers to (see Directory Structure above).