org.apache.flink.runtime.scheduler

类 SchedulerBase

java.lang.Object

org.apache.flink.runtime.scheduler.SchedulerBase

所有已实现的接口:

AutoCloseable, CheckpointScheduling, GlobalFailureHandler, SchedulerNG, org.apache.flink.util.AutoCloseableAsync

直接已知子类:

DefaultScheduler

public abstract class SchedulerBase
extends Object
implements SchedulerNG, CheckpointScheduling

Base class which can be used to implement SchedulerNG.

字段概要

字段

限定符和类型	字段和说明
protected ExecutionVertexVersioner	executionVertexVersioner
protected InputsLocationsRetriever	inputsLocationsRetriever
protected JobManagerJobMetricGroup	jobManagerJobMetricGroup
protected OperatorCoordinatorHandler	operatorCoordinatorHandler
protected StateLocationRetriever	stateLocationRetriever

构造器概要

构造器

构造器和说明

SchedulerBase(org.slf4j.Logger log, JobGraph jobGraph, Executor ioExecutor, org.apache.flink.configuration.Configuration jobMasterConfiguration, CheckpointsCleaner checkpointsCleaner, CheckpointRecoveryFactory checkpointRecoveryFactory, JobManagerJobMetricGroup jobManagerJobMetricGroup, ExecutionVertexVersioner executionVertexVersioner, long initializationTimestamp, org.apache.flink.runtime.concurrent.ComponentMainThreadExecutor mainThreadExecutor, JobStatusListener jobStatusListener, ExecutionGraphFactory executionGraphFactory, VertexParallelismStore vertexParallelismStore)

方法概要

限定符和类型	方法和说明
void	acknowledgeCheckpoint(org.apache.flink.api.common.JobID jobID, ExecutionAttemptID executionAttemptID, long checkpointId, CheckpointMetrics checkpointMetrics, TaskStateSnapshot checkpointState)
protected void	archiveFromFailureHandlingResult(FailureHandlingResultSnapshot failureHandlingResult)
protected void	archiveGlobalFailure(Throwable failure)
void	cancel()
protected abstract void	cancelAllPendingSlotRequestsInternal()
CompletableFuture<Void>	closeAsync()
static VertexParallelismStore	computeVertexParallelismStore(Iterable<JobVertex> vertices) Compute the VertexParallelismStore for all given vertices, which will set defaults and ensure that the returned store contains valid parallelisms.
static VertexParallelismStore	computeVertexParallelismStore(Iterable<JobVertex> vertices, java.util.function.Function<JobVertex,Integer> defaultMaxParallelismFunc)
static VertexParallelismStore	computeVertexParallelismStore(Iterable<JobVertex> vertices, java.util.function.Function<JobVertex,Integer> defaultMaxParallelismFunc, java.util.function.Function<Integer,Integer> normalizeParallelismFunc) Compute the VertexParallelismStore for all given vertices, which will set defaults and ensure that the returned store contains valid parallelisms, with a custom function for default max parallelism calculation and a custom function for normalizing vertex parallelism.
static VertexParallelismStore	computeVertexParallelismStore(JobGraph jobGraph) Compute the VertexParallelismStore for all vertices of a given job graph, which will set defaults and ensure that the returned store contains valid parallelisms.
void	declineCheckpoint(DeclineCheckpoint decline)
CompletableFuture<CoordinationResponse>	deliverCoordinationRequestToCoordinator(OperatorID operator, CoordinationRequest request) Delivers a coordination request to the OperatorCoordinator with the given OperatorID and returns the coordinator's response.
void	deliverOperatorEventToCoordinator(ExecutionAttemptID taskExecutionId, OperatorID operatorId, OperatorEvent evt) Delivers the given OperatorEvent to the OperatorCoordinator with the given OperatorID.
protected void	failJob(Throwable cause, long timestamp)
static int	getDefaultMaxParallelism(JobVertex vertex) Get a default value to use for a given vertex's max parallelism if none was specified.
Iterable<RootExceptionHistoryEntry>	getExceptionHistory()
ExecutionGraph	getExecutionGraph() ExecutionGraph is exposed to make it easier to rework tests to be based on the new scheduler.
ExecutionJobVertex	getExecutionJobVertex(JobVertexID jobVertexId)
ExecutionVertex	getExecutionVertex(ExecutionVertexID executionVertexId)
protected JobGraph	getJobGraph()
CompletableFuture<org.apache.flink.api.common.JobStatus>	getJobTerminationFuture()
protected org.apache.flink.runtime.concurrent.ComponentMainThreadExecutor	getMainThreadExecutor()
protected abstract long	getNumberOfRestarts()
protected ResultPartitionAvailabilityChecker	getResultPartitionAvailabilityChecker()
protected SchedulingTopology	getSchedulingTopology()
void	notifyKvStateRegistered(org.apache.flink.api.common.JobID jobId, JobVertexID jobVertexId, KeyGroupRange keyGroupRange, String registrationName, org.apache.flink.queryablestate.KvStateID kvStateId, InetSocketAddress kvStateServerAddress)
void	notifyKvStateUnregistered(org.apache.flink.api.common.JobID jobId, JobVertexID jobVertexId, KeyGroupRange keyGroupRange, String registrationName)
protected abstract void	onTaskFailed(Execution execution)
protected abstract void	onTaskFinished(Execution execution)
static void	registerJobMetrics(org.apache.flink.metrics.MetricGroup metrics, JobStatusProvider jobStatusProvider, org.apache.flink.metrics.Gauge<Long> numberOfRestarts, DeploymentStateTimeMetrics deploymentTimeMetrics, java.util.function.Consumer<JobStatusListener> jobStatusListenerRegistrar, long initializationTimestamp, org.apache.flink.configuration.MetricOptions.JobStatusMetricsSettings jobStatusMetricsSettings)
void	reportCheckpointMetrics(org.apache.flink.api.common.JobID jobID, ExecutionAttemptID attemptId, long id, CheckpointMetrics metrics)
ExecutionGraphInfo	requestJob()
JobDetails	requestJobDetails()
org.apache.flink.api.common.JobStatus	requestJobStatus()
KvStateLocation	requestKvStateLocation(org.apache.flink.api.common.JobID jobId, String registrationName)
SerializedInputSplit	requestNextInputSplit(JobVertexID vertexID, ExecutionAttemptID executionAttempt)
ExecutionState	requestPartitionState(IntermediateDataSetID intermediateResultId, ResultPartitionID resultPartitionId)
protected void	resetForNewExecution(ExecutionVertexID executionVertexId)
protected void	resetForNewExecutions(Collection<ExecutionVertexID> vertices)
protected void	restoreState(Set<ExecutionVertexID> vertices, boolean isGlobalRecovery)
protected void	setGlobalFailureCause(Throwable cause, long timestamp)
void	startCheckpointScheduler() Starts the periodic scheduling if possible.
void	startScheduling()
protected abstract void	startSchedulingInternal()
void	stopCheckpointScheduler() Stops the periodic scheduling if possible.
CompletableFuture<String>	stopWithSavepoint(String targetDirectory, boolean terminate, org.apache.flink.core.execution.SavepointFormatType formatType)
protected void	transitionExecutionGraphState(org.apache.flink.api.common.JobStatus current, org.apache.flink.api.common.JobStatus newState)
protected void	transitionToRunning()
protected void	transitionToScheduled(List<ExecutionVertexID> verticesToDeploy)
CompletableFuture<String>	triggerCheckpoint()
CompletableFuture<String>	triggerSavepoint(String targetDirectory, boolean cancelJob, org.apache.flink.core.execution.SavepointFormatType formatType)
void	updateAccumulators(AccumulatorSnapshot accumulatorSnapshot)
boolean	updateTaskExecutionState(TaskExecutionStateTransition taskExecutionState)

从类继承的方法 java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

从接口继承的方法 org.apache.flink.runtime.scheduler.SchedulerNG

updateTaskExecutionState

从接口继承的方法 org.apache.flink.runtime.scheduler.GlobalFailureHandler

handleGlobalFailure

从接口继承的方法 org.apache.flink.util.AutoCloseableAsync

close

字段详细资料

stateLocationRetriever

protected final StateLocationRetriever stateLocationRetriever

inputsLocationsRetriever

protected final InputsLocationsRetriever inputsLocationsRetriever

jobManagerJobMetricGroup

protected final JobManagerJobMetricGroup jobManagerJobMetricGroup

executionVertexVersioner

protected final ExecutionVertexVersioner executionVertexVersioner

operatorCoordinatorHandler

protected final OperatorCoordinatorHandler operatorCoordinatorHandler

构造器详细资料

SchedulerBase

public SchedulerBase(org.slf4j.Logger log,
                     JobGraph jobGraph,
                     Executor ioExecutor,
                     org.apache.flink.configuration.Configuration jobMasterConfiguration,
                     CheckpointsCleaner checkpointsCleaner,
                     CheckpointRecoveryFactory checkpointRecoveryFactory,
                     JobManagerJobMetricGroup jobManagerJobMetricGroup,
                     ExecutionVertexVersioner executionVertexVersioner,
                     long initializationTimestamp,
                     org.apache.flink.runtime.concurrent.ComponentMainThreadExecutor mainThreadExecutor,
                     JobStatusListener jobStatusListener,
                     ExecutionGraphFactory executionGraphFactory,
                     VertexParallelismStore vertexParallelismStore)
              throws Exception

抛出:

Exception

方法详细资料

getDefaultMaxParallelism

public static int getDefaultMaxParallelism(JobVertex vertex)

Get a default value to use for a given vertex's max parallelism if none was specified.

参数:

vertex - the vertex to compute a default max parallelism for

返回:

the computed max parallelism

computeVertexParallelismStore

public static VertexParallelismStore computeVertexParallelismStore(Iterable<JobVertex> vertices,
                                                                   java.util.function.Function<JobVertex,Integer> defaultMaxParallelismFunc)

computeVertexParallelismStore

public static VertexParallelismStore computeVertexParallelismStore(Iterable<JobVertex> vertices,
                                                                   java.util.function.Function<JobVertex,Integer> defaultMaxParallelismFunc,
                                                                   java.util.function.Function<Integer,Integer> normalizeParallelismFunc)

Compute the VertexParallelismStore for all given vertices, which will set defaults and ensure that the returned store contains valid parallelisms, with a custom function for default max parallelism calculation and a custom function for normalizing vertex parallelism.

参数:

vertices - the vertices to compute parallelism for

defaultMaxParallelismFunc - a function for computing a default max parallelism if none is specified on a given vertex

normalizeParallelismFunc - a function for normalizing vertex parallelism

返回:

the computed parallelism store

computeVertexParallelismStore

public static VertexParallelismStore computeVertexParallelismStore(Iterable<JobVertex> vertices)

Compute the VertexParallelismStore for all given vertices, which will set defaults and ensure that the returned store contains valid parallelisms.

参数:

vertices - the vertices to compute parallelism for

返回:

the computed parallelism store

computeVertexParallelismStore

public static VertexParallelismStore computeVertexParallelismStore(JobGraph jobGraph)

Compute the VertexParallelismStore for all vertices of a given job graph, which will set defaults and ensure that the returned store contains valid parallelisms.

参数:

jobGraph - the job graph to retrieve vertices from

返回:

the computed parallelism store

resetForNewExecutions

protected void resetForNewExecutions(Collection<ExecutionVertexID> vertices)

resetForNewExecution

protected void resetForNewExecution(ExecutionVertexID executionVertexId)

restoreState

protected void restoreState(Set<ExecutionVertexID> vertices,
                            boolean isGlobalRecovery)
                     throws Exception

抛出:

Exception

transitionToScheduled

protected void transitionToScheduled(List<ExecutionVertexID> verticesToDeploy)

setGlobalFailureCause

protected void setGlobalFailureCause(@Nullable
                                     Throwable cause,
                                     long timestamp)

getMainThreadExecutor

protected org.apache.flink.runtime.concurrent.ComponentMainThreadExecutor getMainThreadExecutor()

failJob

protected void failJob(Throwable cause,
                       long timestamp)

getSchedulingTopology

protected final SchedulingTopology getSchedulingTopology()

getResultPartitionAvailabilityChecker

protected final ResultPartitionAvailabilityChecker getResultPartitionAvailabilityChecker()

transitionToRunning

protected final void transitionToRunning()

getExecutionVertex

public ExecutionVertex getExecutionVertex(ExecutionVertexID executionVertexId)

getExecutionJobVertex

public ExecutionJobVertex getExecutionJobVertex(JobVertexID jobVertexId)

getJobGraph

protected JobGraph getJobGraph()

getNumberOfRestarts

protected abstract long getNumberOfRestarts()

cancelAllPendingSlotRequestsInternal

protected abstract void cancelAllPendingSlotRequestsInternal()

transitionExecutionGraphState

protected void transitionExecutionGraphState(org.apache.flink.api.common.JobStatus current,
                                             org.apache.flink.api.common.JobStatus newState)

getExecutionGraph

@VisibleForTesting
public ExecutionGraph getExecutionGraph()

ExecutionGraph is exposed to make it easier to rework tests to be based on the new scheduler. ExecutionGraph is expected to be used only for state check. Yet at the moment, before all the actions are factored out from ExecutionGraph and its sub-components, some actions may still be performed directly on it.

startScheduling

public final void startScheduling()

指定者:

startScheduling 在接口中 SchedulerNG

registerJobMetrics

public static void registerJobMetrics(org.apache.flink.metrics.MetricGroup metrics,
                                      JobStatusProvider jobStatusProvider,
                                      org.apache.flink.metrics.Gauge<Long> numberOfRestarts,
                                      DeploymentStateTimeMetrics deploymentTimeMetrics,
                                      java.util.function.Consumer<JobStatusListener> jobStatusListenerRegistrar,
                                      long initializationTimestamp,
                                      org.apache.flink.configuration.MetricOptions.JobStatusMetricsSettings jobStatusMetricsSettings)

startSchedulingInternal

protected abstract void startSchedulingInternal()

closeAsync

public CompletableFuture<Void> closeAsync()

指定者:

closeAsync 在接口中 org.apache.flink.util.AutoCloseableAsync

cancel

public void cancel()

指定者:

cancel 在接口中 SchedulerNG

getJobTerminationFuture

public CompletableFuture<org.apache.flink.api.common.JobStatus> getJobTerminationFuture()

指定者:

getJobTerminationFuture 在接口中 SchedulerNG

archiveGlobalFailure

protected final void archiveGlobalFailure(Throwable failure)

archiveFromFailureHandlingResult

protected final void archiveFromFailureHandlingResult(FailureHandlingResultSnapshot failureHandlingResult)

updateTaskExecutionState

public final boolean updateTaskExecutionState(TaskExecutionStateTransition taskExecutionState)

指定者:

updateTaskExecutionState 在接口中 SchedulerNG

onTaskFinished

protected abstract void onTaskFinished(Execution execution)

onTaskFailed

protected abstract void onTaskFailed(Execution execution)

requestNextInputSplit

public SerializedInputSplit requestNextInputSplit(JobVertexID vertexID,
                                                  ExecutionAttemptID executionAttempt)
                                           throws IOException

指定者:

requestNextInputSplit 在接口中 SchedulerNG

抛出:

IOException

requestPartitionState

public ExecutionState requestPartitionState(IntermediateDataSetID intermediateResultId,
                                            ResultPartitionID resultPartitionId)
                                     throws PartitionProducerDisposedException

指定者:

requestPartitionState 在接口中 SchedulerNG

抛出:

PartitionProducerDisposedException

getExceptionHistory

@VisibleForTesting
public Iterable<RootExceptionHistoryEntry> getExceptionHistory()

requestJob

public ExecutionGraphInfo requestJob()

指定者:

requestJob 在接口中 SchedulerNG

requestJobStatus

public org.apache.flink.api.common.JobStatus requestJobStatus()

指定者:

requestJobStatus 在接口中 SchedulerNG

requestJobDetails

public JobDetails requestJobDetails()

指定者:

requestJobDetails 在接口中 SchedulerNG

requestKvStateLocation

public KvStateLocation requestKvStateLocation(org.apache.flink.api.common.JobID jobId,
                                              String registrationName)
                                       throws UnknownKvStateLocation,
                                              FlinkJobNotFoundException

指定者:

requestKvStateLocation 在接口中 SchedulerNG

抛出:

UnknownKvStateLocation

FlinkJobNotFoundException

notifyKvStateRegistered

public void notifyKvStateRegistered(org.apache.flink.api.common.JobID jobId,
                                    JobVertexID jobVertexId,
                                    KeyGroupRange keyGroupRange,
                                    String registrationName,
                                    org.apache.flink.queryablestate.KvStateID kvStateId,
                                    InetSocketAddress kvStateServerAddress)
                             throws FlinkJobNotFoundException

指定者:

notifyKvStateRegistered 在接口中 SchedulerNG

抛出:

FlinkJobNotFoundException

notifyKvStateUnregistered

public void notifyKvStateUnregistered(org.apache.flink.api.common.JobID jobId,
                                      JobVertexID jobVertexId,
                                      KeyGroupRange keyGroupRange,
                                      String registrationName)
                               throws FlinkJobNotFoundException

指定者:

notifyKvStateUnregistered 在接口中 SchedulerNG

抛出:

FlinkJobNotFoundException

updateAccumulators

public void updateAccumulators(AccumulatorSnapshot accumulatorSnapshot)

指定者:

updateAccumulators 在接口中 SchedulerNG

triggerSavepoint

public CompletableFuture<String> triggerSavepoint(String targetDirectory,
                                                  boolean cancelJob,
                                                  org.apache.flink.core.execution.SavepointFormatType formatType)

指定者:

triggerSavepoint 在接口中 SchedulerNG

triggerCheckpoint

public CompletableFuture<String> triggerCheckpoint()

指定者:

triggerCheckpoint 在接口中 SchedulerNG

stopCheckpointScheduler

public void stopCheckpointScheduler()

从接口复制的说明: CheckpointScheduling

Stops the periodic scheduling if possible.

指定者:

stopCheckpointScheduler 在接口中 CheckpointScheduling

startCheckpointScheduler

public void startCheckpointScheduler()

从接口复制的说明: CheckpointScheduling

Starts the periodic scheduling if possible.

指定者:

startCheckpointScheduler 在接口中 CheckpointScheduling

acknowledgeCheckpoint

public void acknowledgeCheckpoint(org.apache.flink.api.common.JobID jobID,
                                  ExecutionAttemptID executionAttemptID,
                                  long checkpointId,
                                  CheckpointMetrics checkpointMetrics,
                                  TaskStateSnapshot checkpointState)

指定者:

acknowledgeCheckpoint 在接口中 SchedulerNG

declineCheckpoint

public void declineCheckpoint(DeclineCheckpoint decline)

指定者:

declineCheckpoint 在接口中 SchedulerNG

reportCheckpointMetrics

public void reportCheckpointMetrics(org.apache.flink.api.common.JobID jobID,
                                    ExecutionAttemptID attemptId,
                                    long id,
                                    CheckpointMetrics metrics)

指定者:

reportCheckpointMetrics 在接口中 SchedulerNG

stopWithSavepoint

public CompletableFuture<String> stopWithSavepoint(@Nullable
                                                   String targetDirectory,
                                                   boolean terminate,
                                                   org.apache.flink.core.execution.SavepointFormatType formatType)

指定者:

stopWithSavepoint 在接口中 SchedulerNG

deliverOperatorEventToCoordinator

public void deliverOperatorEventToCoordinator(ExecutionAttemptID taskExecutionId,
                                              OperatorID operatorId,
                                              OperatorEvent evt)
                                       throws org.apache.flink.util.FlinkException

从接口复制的说明: SchedulerNG

Delivers the given OperatorEvent to the OperatorCoordinator with the given OperatorID.

Failure semantics: If the task manager sends an event for a non-running task or a non-existing operator coordinator, then respond with an exception to the call. If task and coordinator exist, then we assume that the call from the TaskManager was valid, and any bubbling exception needs to cause a job failure

指定者:

deliverOperatorEventToCoordinator 在接口中 SchedulerNG

抛出:

org.apache.flink.util.FlinkException - Thrown, if the task is not running or no operator/coordinator exists for the given ID.

deliverCoordinationRequestToCoordinator

public CompletableFuture<CoordinationResponse> deliverCoordinationRequestToCoordinator(OperatorID operator,
                                                                                       CoordinationRequest request)
                                                                                throws org.apache.flink.util.FlinkException

从接口复制的说明: SchedulerNG

Delivers a coordination request to the OperatorCoordinator with the given OperatorID and returns the coordinator's response.

指定者:

deliverCoordinationRequestToCoordinator 在接口中 SchedulerNG

返回:

A future containing the response.

抛出:

org.apache.flink.util.FlinkException - Thrown, if the task is not running, or no operator/coordinator exists for the given ID, or the coordinator cannot handle client events.