public abstract class AbstractHaServices extends Object implements HighAvailabilityServices
getLeaderNameForResourceManager()
, getLeaderNameForDispatcher()
, getLeaderNameForJobManager(org.apache.flink.api.common.JobID)
, getLeaderNameForRestServer()
. The returned leader name is the ConfigMap name in Kubernetes and
child path in Zookeeper.
close()
and closeAndCleanupAllData()
should be implemented to destroy the
resources.
The abstract class is also responsible for determining which component service should be
reused. For example, runningJobsRegistry
is created once and could be reused many times.
Modifier and Type | Field and Description |
---|---|
protected Configuration |
configuration
The runtime configuration.
|
protected Executor |
ioExecutor
The executor to run external IO operations on.
|
protected org.slf4j.Logger |
logger |
DEFAULT_JOB_ID, DEFAULT_LEADER_ID
Constructor and Description |
---|
AbstractHaServices(Configuration config,
Executor ioExecutor,
BlobStoreService blobStoreService) |
Modifier and Type | Method and Description |
---|---|
void |
cleanupJobData(JobID jobID)
Deletes all data for specified job stored by these services in external stores.
|
void |
close()
Closes the high availability services, releasing all resources.
|
void |
closeAndCleanupAllData()
Closes the high availability services (releasing all resources) and deletes all data stored
by these services in external stores.
|
BlobStore |
createBlobStore()
Creates the BLOB store in which BLOBs are stored in a highly-available fashion.
|
protected abstract CheckpointRecoveryFactory |
createCheckpointRecoveryFactory()
Create the checkpoint recovery factory for the job manager.
|
protected abstract JobGraphStore |
createJobGraphStore()
Create the submitted job graph store for the job manager.
|
protected abstract LeaderElectionService |
createLeaderElectionService(String leaderName)
Create leader election service with specified leaderName.
|
protected abstract LeaderRetrievalService |
createLeaderRetrievalService(String leaderName)
Create leader retrieval service with specified leaderName.
|
protected abstract RunningJobsRegistry |
createRunningJobsRegistry()
Create the registry that holds information about whether jobs are currently running.
|
CheckpointRecoveryFactory |
getCheckpointRecoveryFactory()
Gets the checkpoint recovery factory for the job manager.
|
LeaderElectionService |
getClusterRestEndpointLeaderElectionService()
Gets the leader election service for the cluster's rest endpoint.
|
LeaderRetrievalService |
getClusterRestEndpointLeaderRetriever()
Get the leader retriever for the cluster's rest endpoint.
|
LeaderElectionService |
getDispatcherLeaderElectionService()
Gets the leader election service for the cluster's dispatcher.
|
LeaderRetrievalService |
getDispatcherLeaderRetriever()
Gets the leader retriever for the dispatcher.
|
JobGraphStore |
getJobGraphStore()
Gets the submitted job graph store for the job manager.
|
LeaderElectionService |
getJobManagerLeaderElectionService(JobID jobID)
Gets the leader election service for the given job.
|
LeaderRetrievalService |
getJobManagerLeaderRetriever(JobID jobID)
Gets the leader retriever for the job JobMaster which is responsible for the given job.
|
LeaderRetrievalService |
getJobManagerLeaderRetriever(JobID jobID,
String defaultJobManagerAddress)
Gets the leader retriever for the job JobMaster which is responsible for the given job.
|
protected abstract String |
getLeaderNameForDispatcher()
Get the leader name for Dispatcher.
|
protected abstract String |
getLeaderNameForJobManager(JobID jobID)
Get the leader name for specific JobManager.
|
protected abstract String |
getLeaderNameForResourceManager()
Get the leader name for ResourceManager.
|
protected abstract String |
getLeaderNameForRestServer()
Get the leader name for RestServer.
|
LeaderElectionService |
getResourceManagerLeaderElectionService()
Gets the leader election service for the cluster's resource manager.
|
LeaderRetrievalService |
getResourceManagerLeaderRetriever()
Gets the leader retriever for the cluster's resource manager.
|
RunningJobsRegistry |
getRunningJobsRegistry()
Gets the registry that holds information about whether jobs are currently running.
|
protected abstract void |
internalCleanup()
Clean up the meta data in the distributed system(e.g.
|
protected abstract void |
internalCleanupJobData(JobID jobID)
Clean up the meta data in the distributed system(e.g.
|
protected abstract void |
internalClose()
Closes the components which is used for external operations(e.g.
|
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
getWebMonitorLeaderElectionService, getWebMonitorLeaderRetriever
protected final org.slf4j.Logger logger
protected final Executor ioExecutor
protected final Configuration configuration
public AbstractHaServices(Configuration config, Executor ioExecutor, BlobStoreService blobStoreService)
public LeaderRetrievalService getResourceManagerLeaderRetriever()
HighAvailabilityServices
getResourceManagerLeaderRetriever
in interface HighAvailabilityServices
public LeaderRetrievalService getDispatcherLeaderRetriever()
HighAvailabilityServices
getDispatcherLeaderRetriever
in interface HighAvailabilityServices
public LeaderRetrievalService getJobManagerLeaderRetriever(JobID jobID)
HighAvailabilityServices
getJobManagerLeaderRetriever
in interface HighAvailabilityServices
jobID
- The identifier of the job.public LeaderRetrievalService getJobManagerLeaderRetriever(JobID jobID, String defaultJobManagerAddress)
HighAvailabilityServices
getJobManagerLeaderRetriever
in interface HighAvailabilityServices
jobID
- The identifier of the job.defaultJobManagerAddress
- JobManager address which will be returned by a static leader
retrieval service.public LeaderRetrievalService getClusterRestEndpointLeaderRetriever()
ClientHighAvailabilityServices
getClusterRestEndpointLeaderRetriever
in interface ClientHighAvailabilityServices
getClusterRestEndpointLeaderRetriever
in interface HighAvailabilityServices
public LeaderElectionService getResourceManagerLeaderElectionService()
HighAvailabilityServices
getResourceManagerLeaderElectionService
in interface HighAvailabilityServices
public LeaderElectionService getDispatcherLeaderElectionService()
HighAvailabilityServices
getDispatcherLeaderElectionService
in interface HighAvailabilityServices
public LeaderElectionService getJobManagerLeaderElectionService(JobID jobID)
HighAvailabilityServices
getJobManagerLeaderElectionService
in interface HighAvailabilityServices
jobID
- The identifier of the job running the election.public LeaderElectionService getClusterRestEndpointLeaderElectionService()
HighAvailabilityServices
getClusterRestEndpointLeaderElectionService
in interface HighAvailabilityServices
public CheckpointRecoveryFactory getCheckpointRecoveryFactory()
HighAvailabilityServices
getCheckpointRecoveryFactory
in interface HighAvailabilityServices
public JobGraphStore getJobGraphStore() throws Exception
HighAvailabilityServices
getJobGraphStore
in interface HighAvailabilityServices
Exception
- if the submitted job graph store could not be createdpublic RunningJobsRegistry getRunningJobsRegistry()
HighAvailabilityServices
getRunningJobsRegistry
in interface HighAvailabilityServices
public BlobStore createBlobStore()
HighAvailabilityServices
createBlobStore
in interface HighAvailabilityServices
public void close() throws Exception
HighAvailabilityServices
This method does not delete or clean up any data stored in external stores (file systems, ZooKeeper, etc). Another instance of the high availability services will be able to recover the job.
If an exception occurs during closing services, this method will attempt to continue closing other services and report exceptions only after all services have been attempted to be closed.
close
in interface AutoCloseable
close
in interface HighAvailabilityServices
Exception
- Thrown, if an exception occurred while closing these services.public void closeAndCleanupAllData() throws Exception
HighAvailabilityServices
After this method was called, the any job or session that was managed by these high availability services will be unrecoverable.
If an exception occurs during cleanup, this method will attempt to continue the cleanup and report exceptions only after all cleanup steps have been attempted.
closeAndCleanupAllData
in interface HighAvailabilityServices
Exception
- Thrown, if an exception occurred while closing these services or cleaning
up data stored by them.public void cleanupJobData(JobID jobID) throws Exception
HighAvailabilityServices
cleanupJobData
in interface HighAvailabilityServices
jobID
- The identifier of the job to cleanup.Exception
- Thrown, if an exception occurred while cleaning data stored by them.protected abstract LeaderElectionService createLeaderElectionService(String leaderName)
leaderName
- ConfigMap name in Kubernetes or child node path in Zookeeper.protected abstract LeaderRetrievalService createLeaderRetrievalService(String leaderName)
leaderName
- ConfigMap name in Kubernetes or child node path in Zookeeper.protected abstract CheckpointRecoveryFactory createCheckpointRecoveryFactory()
protected abstract JobGraphStore createJobGraphStore() throws Exception
Exception
- if the submitted job graph store could not be createdprotected abstract RunningJobsRegistry createRunningJobsRegistry()
protected abstract void internalClose()
protected abstract void internalCleanup() throws Exception
If an exception occurs during internal cleanup, we will continue the cleanup in closeAndCleanupAllData()
and report exceptions only after all cleanup steps have been
attempted.
Exception
- when do the cleanup operation on external storage.protected abstract void internalCleanupJobData(JobID jobID) throws Exception
jobID
- The identifier of the job to cleanup.Exception
- when do the cleanup operation on external storage.protected abstract String getLeaderNameForResourceManager()
protected abstract String getLeaderNameForDispatcher()
protected abstract String getLeaderNameForJobManager(JobID jobID)
jobID
- job idprotected abstract String getLeaderNameForRestServer()
Copyright © 2014–2021 The Apache Software Foundation. All rights reserved.