public class FsCheckpointStreamFactory extends Object implements CheckpointStreamFactory
CheckpointStreamFactory
that produces streams that
write to a FileSystem
.
The factory has one core directory into which it puts all checkpoint data. Inside that
directory, it creates a directory per job, inside which each checkpoint gets a directory, with
files for each state, for example:
hdfs://namenode:port/flink-checkpoints/<job-id>/chk-17/6ba7b810-9dad-11d1-80b4-00c04fd430c8
Modifier and Type | Class and Description |
---|---|
static class |
FsCheckpointStreamFactory.FsCheckpointStateOutputStream
A
CheckpointStreamFactory.CheckpointStateOutputStream that writes into a file and
returns a StreamStateHandle upon closing. |
CheckpointStreamFactory.CheckpointStateOutputStream
Constructor and Description |
---|
FsCheckpointStreamFactory(Path checkpointDataUri,
JobID jobId,
int fileStateSizeThreshold)
Creates a new state backend that stores its checkpoint data in the file system and location
defined by the given URI.
|
Modifier and Type | Method and Description |
---|---|
void |
close()
Closes the stream factory, releasing all internal resources, but does not delete any
persistent checkpoint data.
|
protected Path |
createBasePath(FileSystem fs,
Path checkpointDirectory,
JobID jobID) |
protected Path |
createCheckpointDirPath(Path checkpointDirectory,
long checkpointID) |
FsCheckpointStreamFactory.FsCheckpointStateOutputStream |
createCheckpointStateOutputStream(long checkpointID,
long timestamp)
Creates an new
CheckpointStreamFactory.CheckpointStateOutputStream . |
String |
toString() |
public FsCheckpointStreamFactory(Path checkpointDataUri, JobID jobId, int fileStateSizeThreshold) throws IOException
A file system for the file system scheme in the URI (e.g., 'file://', 'hdfs://', or 'S3://')
must be accessible via FileSystem.get(URI)
.
For a state backend targeting HDFS, this means that the URI must either specify the authority (host and port), or that the Hadoop configuration that describes that information must be in the classpath.
checkpointDataUri
- The URI describing the filesystem (scheme and optionally authority),
and the path to the checkpoint data directory.fileStateSizeThreshold
- State up to this size will be stored as part of the metadata,
rather than in filesIOException
- Thrown, if no file system can be found for the scheme in the URI.public void close() throws Exception
CheckpointStreamFactory
close
in interface CheckpointStreamFactory
Exception
- Exceptions can be forwarded and will be logged by the systempublic FsCheckpointStreamFactory.FsCheckpointStateOutputStream createCheckpointStateOutputStream(long checkpointID, long timestamp) throws Exception
CheckpointStreamFactory
CheckpointStreamFactory.CheckpointStateOutputStream
. When the stream
is closed, it returns a state handle that can retrieve the state back.createCheckpointStateOutputStream
in interface CheckpointStreamFactory
checkpointID
- The ID of the checkpoint.timestamp
- The timestamp of the checkpoint.Exception
- Exceptions may occur while creating the stream and should be forwarded.protected Path createBasePath(FileSystem fs, Path checkpointDirectory, JobID jobID) throws IOException
IOException
protected Path createCheckpointDirPath(Path checkpointDirectory, long checkpointID)
Copyright © 2014–2018 The Apache Software Foundation. All rights reserved.