T
- type of the cluster idpublic abstract class ClusterClient<T> extends Object
Modifier and Type | Field and Description |
---|---|
protected Configuration |
flinkConfig
Configuration of the client.
|
protected HighAvailabilityServices |
highAvailabilityServices
Service factory for high available.
|
protected JobExecutionResult |
lastJobExecutionResult
For interactive invocations, the job results are only available after the ContextEnvironment has
been run inside the user JAR.
|
protected org.slf4j.Logger |
log |
protected scala.concurrent.duration.FiniteDuration |
timeout
Timeout for futures.
|
Constructor and Description |
---|
ClusterClient(Configuration flinkConfig)
Creates a instance that submits the programs to the JobManager defined in the
configuration.
|
ClusterClient(Configuration flinkConfig,
HighAvailabilityServices highAvailabilityServices,
boolean sharedHaServices)
Creates a instance that submits the programs to the JobManager defined in the
configuration.
|
Modifier and Type | Method and Description |
---|---|
abstract void |
cancel(JobID jobId)
Cancels a job identified by the job id.
|
abstract String |
cancelWithSavepoint(JobID jobId,
String savepointDirectory)
Cancels a job identified by the job id and triggers a savepoint.
|
abstract CompletableFuture<Acknowledge> |
disposeSavepoint(String savepointPath) |
Map<String,OptionalFailure<Object>> |
getAccumulators(JobID jobID)
Requests and returns the accumulators for the given job identifier.
|
abstract Map<String,OptionalFailure<Object>> |
getAccumulators(JobID jobID,
ClassLoader loader)
Requests and returns the accumulators for the given job identifier.
|
LeaderConnectionInfo |
getClusterConnectionInfo()
Gets the current cluster connection info (may change in case of a HA setup).
|
abstract T |
getClusterId()
Returns the cluster id identifying the cluster to which the client is connected.
|
Configuration |
getFlinkConfiguration()
Return the Flink configuration object.
|
static JobGraph |
getJobGraph(Configuration flinkConfig,
FlinkPlan optPlan,
List<URL> jarFiles,
List<URL> classpaths,
SavepointRestoreSettings savepointSettings) |
static JobGraph |
getJobGraph(Configuration flinkConfig,
PackagedProgram prog,
FlinkPlan optPlan,
SavepointRestoreSettings savepointSettings) |
abstract CompletableFuture<JobStatus> |
getJobStatus(JobID jobId)
|
static FlinkPlan |
getOptimizedPlan(Optimizer compiler,
PackagedProgram prog,
int parallelism) |
static OptimizedPlan |
getOptimizedPlan(Optimizer compiler,
Plan p,
int parallelism) |
static String |
getOptimizedPlanAsJson(Optimizer compiler,
PackagedProgram prog,
int parallelism) |
boolean |
getPrintStatusDuringExecution() |
abstract String |
getWebInterfaceURL()
Returns an URL (as a string) to the JobManager web interface.
|
boolean |
isDetached()
A flag to indicate whether this clients submits jobs detached.
|
abstract CompletableFuture<Collection<JobStatusMessage>> |
listJobs()
Lists the currently running and finished jobs on the cluster.
|
JobSubmissionResult |
run(FlinkPlan compiledPlan,
List<URL> libraries,
List<URL> classpaths,
ClassLoader classLoader) |
JobSubmissionResult |
run(FlinkPlan compiledPlan,
List<URL> libraries,
List<URL> classpaths,
ClassLoader classLoader,
SavepointRestoreSettings savepointSettings) |
JobSubmissionResult |
run(JobWithJars program,
int parallelism) |
JobSubmissionResult |
run(JobWithJars jobWithJars,
int parallelism,
SavepointRestoreSettings savepointSettings)
Runs a program on the Flink cluster to which this client is connected.
|
JobSubmissionResult |
run(PackagedProgram prog,
int parallelism)
General purpose method to run a user jar from the CliFrontend in either blocking or detached mode, depending
on whether
setDetached(true) or setDetached(false) . |
void |
setDetached(boolean isDetached)
Set the mode of this client (detached or blocking job execution).
|
void |
setPrintStatusDuringExecution(boolean print)
Configures whether the client should print progress updates during the execution to
System.out . |
void |
shutdown()
Shuts down the client.
|
void |
shutDownCluster() |
abstract String |
stopWithSavepoint(JobID jobId,
boolean advanceToEndOfEventTime,
String savepointDirectory)
Stops a program on Flink cluster whose job-manager is configured in this client's configuration.
|
abstract JobSubmissionResult |
submitJob(JobGraph jobGraph,
ClassLoader classLoader)
Calls the subclasses' submitJob method.
|
abstract CompletableFuture<String> |
triggerSavepoint(JobID jobId,
String savepointDirectory)
Triggers a savepoint for the job identified by the job id.
|
protected final org.slf4j.Logger log
protected final Configuration flinkConfig
protected final scala.concurrent.duration.FiniteDuration timeout
protected final HighAvailabilityServices highAvailabilityServices
protected JobExecutionResult lastJobExecutionResult
public ClusterClient(Configuration flinkConfig) throws Exception
flinkConfig
- The config used to obtain the job-manager's address, and used to configure the optimizer.Exception
- we cannot create the high availability servicespublic ClusterClient(Configuration flinkConfig, HighAvailabilityServices highAvailabilityServices, boolean sharedHaServices)
flinkConfig
- The config used to obtain the job-manager's address, and used to configure the optimizer.highAvailabilityServices
- HighAvailabilityServices to use for leader retrievalsharedHaServices
- true if the HighAvailabilityServices are shared and must not be shut downpublic void shutdown() throws Exception
Exception
public void setPrintStatusDuringExecution(boolean print)
System.out
.
All updates are logged via the SLF4J loggers regardless of this setting.print
- True to print updates to standard out during execution, false to not print them.public boolean getPrintStatusDuringExecution()
System.out
public LeaderConnectionInfo getClusterConnectionInfo() throws LeaderRetrievalException
LeaderRetrievalException
- if the leader could not be retrievedpublic static String getOptimizedPlanAsJson(Optimizer compiler, PackagedProgram prog, int parallelism) throws CompilerException, ProgramInvocationException
public static FlinkPlan getOptimizedPlan(Optimizer compiler, PackagedProgram prog, int parallelism) throws CompilerException, ProgramInvocationException
public static OptimizedPlan getOptimizedPlan(Optimizer compiler, Plan p, int parallelism) throws CompilerException
CompilerException
public JobSubmissionResult run(PackagedProgram prog, int parallelism) throws ProgramInvocationException, ProgramMissingJobException
setDetached(true)
or setDetached(false)
.prog
- the packaged programparallelism
- the parallelism to execute the contained Flink jobProgramMissingJobException
ProgramInvocationException
public JobSubmissionResult run(JobWithJars program, int parallelism) throws ProgramInvocationException
ProgramInvocationException
public JobSubmissionResult run(JobWithJars jobWithJars, int parallelism, SavepointRestoreSettings savepointSettings) throws CompilerException, ProgramInvocationException
jobWithJars
- The program to be executed.parallelism
- The default parallelism to use when running the program. The default parallelism is used
when the program does not set a parallelism by itself.CompilerException
- Thrown, if the compiler encounters an illegal situation.ProgramInvocationException
- Thrown, if the program could not be instantiated from its jar file,
or if the submission failed. That might be either due to an I/O problem,
i.e. the job-manager is unreachable, or due to the fact that the
parallel execution failed.public JobSubmissionResult run(FlinkPlan compiledPlan, List<URL> libraries, List<URL> classpaths, ClassLoader classLoader) throws ProgramInvocationException
ProgramInvocationException
public JobSubmissionResult run(FlinkPlan compiledPlan, List<URL> libraries, List<URL> classpaths, ClassLoader classLoader, SavepointRestoreSettings savepointSettings) throws ProgramInvocationException
ProgramInvocationException
public abstract CompletableFuture<JobStatus> getJobStatus(JobID jobId)
public abstract void cancel(JobID jobId) throws Exception
jobId
- the job idException
- In case an error occurred.public abstract String cancelWithSavepoint(JobID jobId, @Nullable String savepointDirectory) throws Exception
jobId
- the job idsavepointDirectory
- directory the savepoint should be written toException
- In case an error occurred.public abstract String stopWithSavepoint(JobID jobId, boolean advanceToEndOfEventTime, @Nullable String savepointDirectory) throws Exception
jobId
- the job ID of the streaming program to stopadvanceToEndOfEventTime
- flag indicating if the source should inject a MAX_WATERMARK
in the pipelinesavepointDirectory
- directory the savepoint should be written toCompletableFuture
containing the path where the savepoint is locatedException
- If the job ID is invalid (ie, is unknown or refers to a batch job) or if sending the stop signal
failed. That might be due to an I/O problem, ie, the job-manager is unreachable.public abstract CompletableFuture<String> triggerSavepoint(JobID jobId, @Nullable String savepointDirectory) throws FlinkException
CheckpointingOptions.SAVEPOINT_DIRECTORY
if it is null.jobId
- job idsavepointDirectory
- directory the savepoint should be written toFlinkException
- if no connection to the cluster could be establishedpublic abstract CompletableFuture<Acknowledge> disposeSavepoint(String savepointPath) throws FlinkException
FlinkException
public abstract CompletableFuture<Collection<JobStatusMessage>> listJobs() throws Exception
Exception
- if no connection to the cluster could be establishedpublic Map<String,OptionalFailure<Object>> getAccumulators(JobID jobID) throws Exception
jobID
- The job identifier of a job.Exception
public abstract Map<String,OptionalFailure<Object>> getAccumulators(JobID jobID, ClassLoader loader) throws Exception
jobID
- The job identifier of a job.loader
- The class loader for deserializing the accumulator results.Exception
public static JobGraph getJobGraph(Configuration flinkConfig, PackagedProgram prog, FlinkPlan optPlan, SavepointRestoreSettings savepointSettings) throws ProgramInvocationException
ProgramInvocationException
public static JobGraph getJobGraph(Configuration flinkConfig, FlinkPlan optPlan, List<URL> jarFiles, List<URL> classpaths, SavepointRestoreSettings savepointSettings)
public abstract String getWebInterfaceURL()
public abstract T getClusterId()
public void setDetached(boolean isDetached)
isDetached
- If true, the client will submit programs detached via the run
methodpublic boolean isDetached()
public Configuration getFlinkConfiguration()
public abstract JobSubmissionResult submitJob(JobGraph jobGraph, ClassLoader classLoader) throws ProgramInvocationException
jobGraph
- The JobGraph to be submittedProgramInvocationException
public void shutDownCluster()
Copyright © 2014–2020 The Apache Software Foundation. All rights reserved.