public class ZooKeeperCompletedCheckpointStore extends Object implements CompletedCheckpointStore
CompletedCheckpointStore
for JobManagers running in HighAvailabilityMode.ZOOKEEPER
.
Checkpoints are added under a ZNode per job:
+----O /flink/checkpoints/<job-id> [persistent] . | . +----O /flink/checkpoints/<job-id>/1 [persistent] . . . . . . . . . . +----O /flink/checkpoints/<job-id>/N [persistent]
During recovery, the latest checkpoint is read from ZooKeeper. If there is more than one, only the latest one is used and older ones are discarded (even if the maximum number of retained checkpoints is greater than one).
If there is a network partition and multiple JobManagers run concurrent checkpoints for the same program, it is OK to take any valid successful checkpoint as long as the "history" of checkpoints is consistent. Currently, after recovery we start out with only a single checkpoint to circumvent those situations.
Constructor and Description |
---|
ZooKeeperCompletedCheckpointStore(int maxNumberOfCheckpointsToRetain,
org.apache.curator.framework.CuratorFramework client,
String checkpointsPath,
RetrievableStateStorageHelper<CompletedCheckpoint> stateStorage,
Executor executor)
Creates a
ZooKeeperCompletedCheckpointStore instance. |
Modifier and Type | Method and Description |
---|---|
void |
addCheckpoint(CompletedCheckpoint checkpoint)
Synchronously writes the new checkpoints to ZooKeeper and asynchronously removes older ones.
|
static String |
checkpointIdToPath(long checkpointId)
Convert a checkpoint id into a ZooKeeper path.
|
List<CompletedCheckpoint> |
getAllCheckpoints()
Returns all
CompletedCheckpoint instances. |
CompletedCheckpoint |
getLatestCheckpoint()
Returns the latest
CompletedCheckpoint instance or null if none was
added. |
int |
getMaxNumberOfRetainedCheckpoints()
Returns the max number of retained checkpoints.
|
int |
getNumberOfRetainedCheckpoints()
Returns the current number of retained checkpoints.
|
static long |
pathToCheckpointId(String path)
Converts a path to the checkpoint id.
|
void |
recover()
Gets the latest checkpoint from ZooKeeper and removes all others.
|
boolean |
requiresExternalizedCheckpoints()
This method returns whether the completed checkpoint store requires checkpoints to be
externalized.
|
void |
shutdown(JobStatus jobStatus)
Shuts down the store.
|
public ZooKeeperCompletedCheckpointStore(int maxNumberOfCheckpointsToRetain, org.apache.curator.framework.CuratorFramework client, String checkpointsPath, RetrievableStateStorageHelper<CompletedCheckpoint> stateStorage, Executor executor) throws Exception
ZooKeeperCompletedCheckpointStore
instance.maxNumberOfCheckpointsToRetain
- The maximum number of checkpoints to retain (at
least 1). Adding more checkpoints than this results
in older checkpoints being discarded. On recovery,
we will only start with a single checkpoint.client
- The Curator ZooKeeper clientcheckpointsPath
- The ZooKeeper path for the checkpoints (needs to
start with a '/')stateStorage
- State storage to be used to persist the completed
checkpointexecutor
- to execute blocking callsException
public boolean requiresExternalizedCheckpoints()
CompletedCheckpointStore
requiresExternalizedCheckpoints
in interface CompletedCheckpointStore
public void recover() throws Exception
Important: Even if there are more than one checkpoint in ZooKeeper, this will only recover the latest and discard the others. Otherwise, there is no guarantee that the history of checkpoints is consistent.
recover
in interface CompletedCheckpointStore
Exception
public void addCheckpoint(CompletedCheckpoint checkpoint) throws Exception
addCheckpoint
in interface CompletedCheckpointStore
checkpoint
- Completed checkpoint to add.Exception
public CompletedCheckpoint getLatestCheckpoint()
CompletedCheckpointStore
CompletedCheckpoint
instance or null
if none was
added.getLatestCheckpoint
in interface CompletedCheckpointStore
public List<CompletedCheckpoint> getAllCheckpoints() throws Exception
CompletedCheckpointStore
CompletedCheckpoint
instances.
Returns an empty list if no checkpoint has been added yet.
getAllCheckpoints
in interface CompletedCheckpointStore
Exception
public int getNumberOfRetainedCheckpoints()
CompletedCheckpointStore
getNumberOfRetainedCheckpoints
in interface CompletedCheckpointStore
public int getMaxNumberOfRetainedCheckpoints()
CompletedCheckpointStore
getMaxNumberOfRetainedCheckpoints
in interface CompletedCheckpointStore
public void shutdown(JobStatus jobStatus) throws Exception
CompletedCheckpointStore
The job status is forwarded and used to decide whether state should actually be discarded or kept.
shutdown
in interface CompletedCheckpointStore
jobStatus
- Job state on shut downException
public static String checkpointIdToPath(long checkpointId)
checkpointId
- to convert to the pathpublic static long pathToCheckpointId(String path)
path
- in ZooKeeperCopyright © 2014–2019 The Apache Software Foundation. All rights reserved.