public abstract class YarnHighAvailabilityServices extends Object implements HighAvailabilityServices
HighAvailabilityServices
for YARN setups.
These high-availability services auto-configure YARN's HDFS and the YARN application's
working directory to be used to store job recovery data.
Note for implementers: This class locks access to and creation of services,
to make sure all services are properly shut down when shutting down this class.
To participate in the checks, overriding methods should frame method body with
calls to enter()
and exit()
as shown in the following pattern:
public LeaderRetrievalService getResourceManagerLeaderRetriever() {
enter();
try {
CuratorClient client = getCuratorClient();
return ZooKeeperUtils.createLeaderRetrievalService(client, configuration, RESOURCE_MANAGER_LEADER_PATH);
} finally {
exit();
}
}
Modifier and Type | Field and Description |
---|---|
protected BlobStoreService |
blobStoreService
Blob store service to be used for the BlobServer and BlobCache.
|
static String |
FLINK_RECOVERY_DATA_DIR
The name of the sub directory in which Flink stores the recovery data.
|
protected FileSystem |
flinkFileSystem
The Flink FileSystem object that represent the HDFS used by YARN.
|
protected Path |
haDataDirectory
The directory for HA persistent data.
|
protected org.apache.hadoop.fs.FileSystem |
hadoopFileSystem
The Hadoop FileSystem object that represent the HDFS used by YARN.
|
protected static org.slf4j.Logger |
LOG
Logger for these services, shared with subclasses.
|
protected Path |
workingDirectory
The working directory of this YARN application.
|
DEFAULT_JOB_ID, DEFAULT_LEADER_ID
Modifier | Constructor and Description |
---|---|
protected |
YarnHighAvailabilityServices(Configuration config,
Configuration hadoopConf)
Creates new YARN high-availability services, configuring the file system and recovery
data directory based on the working directory in the given Hadoop configuration.
|
Modifier and Type | Method and Description |
---|---|
void |
close()
Closes the high availability services, releasing all resources.
|
void |
closeAndCleanupAllData()
Closes the high availability services (releasing all resources) and deletes
all data stored by these services in external stores.
|
BlobStore |
createBlobStore()
Creates the BLOB store in which BLOBs are stored in a highly-available fashion.
|
static YarnHighAvailabilityServices |
forSingleJobAppMaster(Configuration flinkConfig,
Configuration hadoopConfig)
Creates the high-availability services for a single-job Flink YARN application, to be
used in the Application Master that runs both ResourceManager and JobManager.
|
static YarnHighAvailabilityServices |
forYarnTaskManager(Configuration flinkConfig,
Configuration hadoopConfig)
Creates the high-availability services for the TaskManagers participating in
a Flink YARN application.
|
boolean |
isClosed()
Checks whether these services have been shut down.
|
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
getCheckpointRecoveryFactory, getDispatcherLeaderElectionService, getDispatcherLeaderRetriever, getJobManagerLeaderElectionService, getJobManagerLeaderRetriever, getJobManagerLeaderRetriever, getResourceManagerLeaderElectionService, getResourceManagerLeaderRetriever, getRunningJobsRegistry, getSubmittedJobGraphStore, getWebMonitorLeaderElectionService, getWebMonitorLeaderRetriever
public static final String FLINK_RECOVERY_DATA_DIR
protected static final org.slf4j.Logger LOG
protected final FileSystem flinkFileSystem
protected final org.apache.hadoop.fs.FileSystem hadoopFileSystem
protected final Path workingDirectory
protected final Path haDataDirectory
protected final BlobStoreService blobStoreService
protected YarnHighAvailabilityServices(Configuration config, Configuration hadoopConf) throws IOException
This class requires that the default Hadoop file system configured in the given Hadoop configuration is an HDFS.
config
- The Flink configuration of this component / process.hadoopConf
- The Hadoop configuration for the YARN cluster.IOException
- Thrown, if the initialization of the Hadoop file system used by YARN fails.public BlobStore createBlobStore() throws IOException
HighAvailabilityServices
createBlobStore
in interface HighAvailabilityServices
IOException
- if the blob store could not be createdpublic boolean isClosed()
public void close() throws Exception
HighAvailabilityServices
This method does not delete or clean up any data stored in external stores (file systems, ZooKeeper, etc). Another instance of the high availability services will be able to recover the job.
If an exception occurs during closing services, this method will attempt to continue closing other services and report exceptions only after all services have been attempted to be closed.
close
in interface AutoCloseable
close
in interface HighAvailabilityServices
Exception
- Thrown, if an exception occurred while closing these services.public void closeAndCleanupAllData() throws Exception
HighAvailabilityServices
After this method was called, the any job or session that was managed by these high availability services will be unrecoverable.
If an exception occurs during cleanup, this method will attempt to continue the cleanup and report exceptions only after all cleanup steps have been attempted.
closeAndCleanupAllData
in interface HighAvailabilityServices
Exception
- Thrown, if an exception occurred while closing these services
or cleaning up data stored by them.public static YarnHighAvailabilityServices forSingleJobAppMaster(Configuration flinkConfig, Configuration hadoopConfig) throws IOException
flinkConfig
- The Flink configuration.hadoopConfig
- The Hadoop configuration for the YARN cluster.IOException
- Thrown, if the high-availability services could not be initialized.public static YarnHighAvailabilityServices forYarnTaskManager(Configuration flinkConfig, Configuration hadoopConfig) throws IOException
flinkConfig
- The Flink configuration.hadoopConfig
- The Hadoop configuration for the YARN cluster.IOException
- Thrown, if the high-availability services could not be initialized.Copyright © 2014–2020 The Apache Software Foundation. All rights reserved.