public abstract class SchedulerBase extends Object implements SchedulerNG, CheckpointScheduling
SchedulerNG
.Modifier and Type | Field and Description |
---|---|
protected ExecutionVertexVersioner |
executionVertexVersioner |
protected InputsLocationsRetriever |
inputsLocationsRetriever |
protected StateLocationRetriever |
stateLocationRetriever |
Constructor and Description |
---|
SchedulerBase(org.slf4j.Logger log,
JobGraph jobGraph,
Executor ioExecutor,
Configuration jobMasterConfiguration,
ClassLoader userCodeLoader,
CheckpointRecoveryFactory checkpointRecoveryFactory,
JobManagerJobMetricGroup jobManagerJobMetricGroup,
ExecutionVertexVersioner executionVertexVersioner,
long initializationTimestamp,
ComponentMainThreadExecutor mainThreadExecutor,
JobStatusListener jobStatusListener,
ExecutionGraphFactory executionGraphFactory) |
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
handleGlobalFailure, updateTaskExecutionState
close
protected final StateLocationRetriever stateLocationRetriever
protected final InputsLocationsRetriever inputsLocationsRetriever
protected final ExecutionVertexVersioner executionVertexVersioner
public SchedulerBase(org.slf4j.Logger log, JobGraph jobGraph, Executor ioExecutor, Configuration jobMasterConfiguration, ClassLoader userCodeLoader, CheckpointRecoveryFactory checkpointRecoveryFactory, JobManagerJobMetricGroup jobManagerJobMetricGroup, ExecutionVertexVersioner executionVertexVersioner, long initializationTimestamp, ComponentMainThreadExecutor mainThreadExecutor, JobStatusListener jobStatusListener, ExecutionGraphFactory executionGraphFactory) throws Exception
Exception
public static int getDefaultMaxParallelism(JobVertex vertex)
vertex
- the vertex to compute a default max parallelism forpublic static VertexParallelismStore computeVertexParallelismStore(Iterable<JobVertex> vertices, java.util.function.Function<JobVertex,Integer> defaultMaxParallelismFunc)
VertexParallelismStore
for all given vertices, which will set defaults
and ensure that the returned store contains valid parallelisms, with a custom function for
default max parallelism calculation.vertices
- the vertices to compute parallelism fordefaultMaxParallelismFunc
- a function for computing a default max parallelism if none
is specified on a given vertexpublic static VertexParallelismStore computeVertexParallelismStore(Iterable<JobVertex> vertices)
VertexParallelismStore
for all given vertices, which will set defaults
and ensure that the returned store contains valid parallelisms.vertices
- the vertices to compute parallelism forpublic static VertexParallelismStore computeVertexParallelismStore(JobGraph jobGraph)
VertexParallelismStore
for all vertices of a given job graph, which will
set defaults and ensure that the returned store contains valid parallelisms.jobGraph
- the job graph to retrieve vertices fromprotected void resetForNewExecutions(Collection<ExecutionVertexID> vertices)
protected void restoreState(Set<ExecutionVertexID> vertices, boolean isGlobalRecovery) throws Exception
Exception
protected void transitionToScheduled(List<ExecutionVertexID> verticesToDeploy)
protected void setGlobalFailureCause(@Nullable Throwable cause, long timestamp)
protected ComponentMainThreadExecutor getMainThreadExecutor()
protected void failJob(Throwable cause, long timestamp)
protected final SchedulingTopology getSchedulingTopology()
protected final ResultPartitionAvailabilityChecker getResultPartitionAvailabilityChecker()
protected final void transitionToRunning()
protected Optional<ExecutionVertexID> getExecutionVertexId(ExecutionAttemptID executionAttemptId)
protected ExecutionVertexID getExecutionVertexIdOrThrow(ExecutionAttemptID executionAttemptId)
public ExecutionVertex getExecutionVertex(ExecutionVertexID executionVertexId)
public ExecutionJobVertex getExecutionJobVertex(JobVertexID jobVertexId)
protected JobGraph getJobGraph()
protected abstract long getNumberOfRestarts()
protected abstract void cancelAllPendingSlotRequestsInternal()
protected void transitionExecutionGraphState(JobStatus current, JobStatus newState)
@VisibleForTesting public ExecutionGraph getExecutionGraph()
public final void startScheduling()
startScheduling
in interface SchedulerNG
protected abstract void startSchedulingInternal()
public CompletableFuture<Void> closeAsync()
AutoCloseableAsync
closeAsync
in interface AutoCloseableAsync
public void cancel()
cancel
in interface SchedulerNG
public CompletableFuture<JobStatus> getJobTerminationFuture()
getJobTerminationFuture
in interface SchedulerNG
protected final void archiveGlobalFailure(Throwable failure)
protected final void archiveFromFailureHandlingResult(FailureHandlingResultSnapshot failureHandlingResult)
public final boolean updateTaskExecutionState(TaskExecutionStateTransition taskExecutionState)
updateTaskExecutionState
in interface SchedulerNG
protected void updateTaskExecutionStateInternal(ExecutionVertexID executionVertexId, TaskExecutionStateTransition taskExecutionState)
public SerializedInputSplit requestNextInputSplit(JobVertexID vertexID, ExecutionAttemptID executionAttempt) throws IOException
requestNextInputSplit
in interface SchedulerNG
IOException
public ExecutionState requestPartitionState(IntermediateDataSetID intermediateResultId, ResultPartitionID resultPartitionId) throws PartitionProducerDisposedException
requestPartitionState
in interface SchedulerNG
PartitionProducerDisposedException
public final void notifyPartitionDataAvailable(ResultPartitionID partitionId)
notifyPartitionDataAvailable
in interface SchedulerNG
protected void notifyPartitionDataAvailableInternal(IntermediateResultPartitionID resultPartitionId)
public ExecutionGraphInfo requestJob()
requestJob
in interface SchedulerNG
public JobStatus requestJobStatus()
requestJobStatus
in interface SchedulerNG
public JobDetails requestJobDetails()
requestJobDetails
in interface SchedulerNG
public KvStateLocation requestKvStateLocation(JobID jobId, String registrationName) throws UnknownKvStateLocation, FlinkJobNotFoundException
requestKvStateLocation
in interface SchedulerNG
UnknownKvStateLocation
FlinkJobNotFoundException
public void notifyKvStateRegistered(JobID jobId, JobVertexID jobVertexId, KeyGroupRange keyGroupRange, String registrationName, KvStateID kvStateId, InetSocketAddress kvStateServerAddress) throws FlinkJobNotFoundException
notifyKvStateRegistered
in interface SchedulerNG
FlinkJobNotFoundException
public void notifyKvStateUnregistered(JobID jobId, JobVertexID jobVertexId, KeyGroupRange keyGroupRange, String registrationName) throws FlinkJobNotFoundException
notifyKvStateUnregistered
in interface SchedulerNG
FlinkJobNotFoundException
public void updateAccumulators(AccumulatorSnapshot accumulatorSnapshot)
updateAccumulators
in interface SchedulerNG
public CompletableFuture<String> triggerSavepoint(String targetDirectory, boolean cancelJob)
triggerSavepoint
in interface SchedulerNG
public void stopCheckpointScheduler()
CheckpointScheduling
stopCheckpointScheduler
in interface CheckpointScheduling
public void startCheckpointScheduler()
CheckpointScheduling
startCheckpointScheduler
in interface CheckpointScheduling
public void acknowledgeCheckpoint(JobID jobID, ExecutionAttemptID executionAttemptID, long checkpointId, CheckpointMetrics checkpointMetrics, TaskStateSnapshot checkpointState)
acknowledgeCheckpoint
in interface SchedulerNG
public void declineCheckpoint(DeclineCheckpoint decline)
declineCheckpoint
in interface SchedulerNG
public void reportCheckpointMetrics(JobID jobID, ExecutionAttemptID attemptId, long id, CheckpointMetrics metrics)
reportCheckpointMetrics
in interface SchedulerNG
public CompletableFuture<String> stopWithSavepoint(@Nullable String targetDirectory, boolean terminate)
stopWithSavepoint
in interface SchedulerNG
public void deliverOperatorEventToCoordinator(ExecutionAttemptID taskExecutionId, OperatorID operatorId, OperatorEvent evt) throws FlinkException
SchedulerNG
OperatorCoordinator
with the given OperatorID
.
Failure semantics: If the task manager sends an event for a non-running task or a non-existing operator coordinator, then respond with an exception to the call. If task and coordinator exist, then we assume that the call from the TaskManager was valid, and any bubbling exception needs to cause a job failure
deliverOperatorEventToCoordinator
in interface SchedulerNG
FlinkException
- Thrown, if the task is not running or no operator/coordinator exists
for the given ID.public CompletableFuture<CoordinationResponse> deliverCoordinationRequestToCoordinator(OperatorID operator, CoordinationRequest request) throws FlinkException
SchedulerNG
OperatorCoordinator
with the given OperatorID
and returns the coordinator's response.deliverCoordinationRequestToCoordinator
in interface SchedulerNG
FlinkException
- Thrown, if the task is not running, or no operator/coordinator exists
for the given ID, or the coordinator cannot handle client events.Copyright © 2014–2022 The Apache Software Foundation. All rights reserved.