public class DefaultCompletedCheckpointStore<R extends ResourceVersion<R>> extends Object implements CompletedCheckpointStore
CompletedCheckpointStore
. Combined with different StateHandleStore
, we could persist the completed
checkpoints to various storage.
During recovery, the latest checkpoint is read from StateHandleStore
. If there is more
than one, only the latest one is used and older ones are discarded (even if the maximum number of
retained checkpoints is greater than one).
If there is a network partition and multiple JobManagers run concurrent checkpoints for the same program, it is OK to take any valid successful checkpoint as long as the "history" of checkpoints is consistent. Currently, after recovery we start out with only a single checkpoint to circumvent those situations.
Constructor and Description |
---|
DefaultCompletedCheckpointStore(int maxNumberOfCheckpointsToRetain,
StateHandleStore<CompletedCheckpoint,R> stateHandleStore,
CheckpointStoreUtil completedCheckpointStoreUtil,
Executor executor)
Creates a
DefaultCompletedCheckpointStore instance. |
Modifier and Type | Method and Description |
---|---|
void |
addCheckpoint(CompletedCheckpoint checkpoint,
CheckpointsCleaner checkpointsCleaner,
Runnable postCleanup)
Synchronously writes the new checkpoints to state handle store and asynchronously removes
older ones.
|
List<CompletedCheckpoint> |
getAllCheckpoints()
Returns all
CompletedCheckpoint instances. |
int |
getMaxNumberOfRetainedCheckpoints()
Returns the max number of retained checkpoints.
|
int |
getNumberOfRetainedCheckpoints()
Returns the current number of retained checkpoints.
|
void |
recover()
Recover all the valid checkpoints from state handle store.
|
boolean |
requiresExternalizedCheckpoints()
This method returns whether the completed checkpoint store requires checkpoints to be
externalized.
|
void |
shutdown(JobStatus jobStatus,
CheckpointsCleaner checkpointsCleaner,
Runnable postCleanup)
Shuts down the store.
|
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
getLatestCheckpoint, storeFor
public DefaultCompletedCheckpointStore(int maxNumberOfCheckpointsToRetain, StateHandleStore<CompletedCheckpoint,R> stateHandleStore, CheckpointStoreUtil completedCheckpointStoreUtil, Executor executor)
DefaultCompletedCheckpointStore
instance.maxNumberOfCheckpointsToRetain
- The maximum number of checkpoints to retain (at least
1). Adding more checkpoints than this results in older checkpoints being discarded. On
recovery, we will only start with a single checkpoint.stateHandleStore
- Completed checkpoints in external storecompletedCheckpointStoreUtil
- utilities for completed checkpoint storeexecutor
- to execute blocking callspublic boolean requiresExternalizedCheckpoints()
CompletedCheckpointStore
requiresExternalizedCheckpoints
in interface CompletedCheckpointStore
public void recover() throws Exception
completedCheckpoints
sorted by checkpoint id.recover
in interface CompletedCheckpointStore
Exception
public void addCheckpoint(CompletedCheckpoint checkpoint, CheckpointsCleaner checkpointsCleaner, Runnable postCleanup) throws Exception
addCheckpoint
in interface CompletedCheckpointStore
checkpoint
- Completed checkpoint to add.PossibleInconsistentStateException
- if adding the checkpoint failed and leaving the
system in an possibly inconsistent state, i.e. it's uncertain whether the checkpoint
metadata was fully written to the underlying systems or not.Exception
public List<CompletedCheckpoint> getAllCheckpoints()
CompletedCheckpointStore
CompletedCheckpoint
instances.
Returns an empty list if no checkpoint has been added yet.
getAllCheckpoints
in interface CompletedCheckpointStore
public int getNumberOfRetainedCheckpoints()
CompletedCheckpointStore
getNumberOfRetainedCheckpoints
in interface CompletedCheckpointStore
public int getMaxNumberOfRetainedCheckpoints()
CompletedCheckpointStore
getMaxNumberOfRetainedCheckpoints
in interface CompletedCheckpointStore
public void shutdown(JobStatus jobStatus, CheckpointsCleaner checkpointsCleaner, Runnable postCleanup) throws Exception
CompletedCheckpointStore
The job status is forwarded and used to decide whether state should actually be discarded or kept.
shutdown
in interface CompletedCheckpointStore
jobStatus
- Job state on shut downException
Copyright © 2014–2021 The Apache Software Foundation. All rights reserved.