org.apache.flink.runtime.scheduler.adaptive

类 AdaptiveScheduler

java.lang.Object

org.apache.flink.runtime.scheduler.adaptive.AdaptiveScheduler

所有已实现的接口:

AutoCloseable, GlobalFailureHandler, SchedulerNG, org.apache.flink.util.AutoCloseableAsync

public class AdaptiveScheduler
extends Object
implements SchedulerNG

A SchedulerNG implementation that uses the declarative resource management and automatically adapts the parallelism in case not enough resource could be acquired to run at the configured parallelism, as described in FLIP-160.

This scheduler only supports jobs with streaming semantics, i.e., all vertices are connected via pipelined data-exchanges.

The implementation is spread over multiple State classes that control which RPCs are allowed in a given state and what state transitions are possible (see the FLIP for an overview). This class can thus be roughly split into 2 parts:

1) RPCs, which must forward the call to the state via State.tryRun(Class, ThrowingConsumer, String) or State.tryCall(Class, FunctionWithException, String).

2) Context methods, which are called by states, to either transition into another state or access functionality of some component in the scheduler.

构造器概要

构造器

构造器和说明

AdaptiveScheduler(JobGraph jobGraph, org.apache.flink.configuration.Configuration configuration, DeclarativeSlotPool declarativeSlotPool, SlotAllocator slotAllocator, Executor ioExecutor, ClassLoader userCodeClassLoader, CheckpointsCleaner checkpointsCleaner, CheckpointRecoveryFactory checkpointRecoveryFactory, java.time.Duration initialResourceAllocationTimeout, java.time.Duration resourceStabilizationTimeout, JobManagerJobMetricGroup jobManagerJobMetricGroup, RestartBackoffTimeStrategy restartBackoffTimeStrategy, long initializationTimestamp, org.apache.flink.runtime.concurrent.ComponentMainThreadExecutor mainThreadExecutor, org.apache.flink.runtime.rpc.FatalErrorHandler fatalErrorHandler, JobStatusListener jobStatusListener, ExecutionGraphFactory executionGraphFactory)

方法概要

限定符和类型	方法和说明
void	acknowledgeCheckpoint(org.apache.flink.api.common.JobID jobID, ExecutionAttemptID executionAttemptID, long checkpointId, CheckpointMetrics checkpointMetrics, TaskStateSnapshot checkpointState)
void	archiveFailure(RootExceptionHistoryEntry failure) Archive failure.
void	cancel()
boolean	canScaleUp(ExecutionGraph executionGraph) Asks if we can scale up the currently executing job.
CompletableFuture<Void>	closeAsync()
void	declineCheckpoint(DeclineCheckpoint decline)
CompletableFuture<CoordinationResponse>	deliverCoordinationRequestToCoordinator(OperatorID operator, CoordinationRequest request) Delivers a coordination request to the OperatorCoordinator with the given OperatorID and returns the coordinator's response.
void	deliverOperatorEventToCoordinator(ExecutionAttemptID taskExecution, OperatorID operator, OperatorEvent evt) Delivers the given OperatorEvent to the OperatorCoordinator with the given OperatorID.
ArchivedExecutionGraph	getArchivedExecutionGraph(org.apache.flink.api.common.JobStatus jobStatus, Throwable cause) Creates an ArchivedExecutionGraph for the given jobStatus and failure cause.
Executor	getIOExecutor() Gets the I/O executor.
CompletableFuture<org.apache.flink.api.common.JobStatus>	getJobTerminationFuture()
org.apache.flink.runtime.concurrent.ComponentMainThreadExecutor	getMainThreadExecutor() Gets the main thread executor.
void	goToCanceling(ExecutionGraph executionGraph, ExecutionGraphHandler executionGraphHandler, OperatorCoordinatorHandler operatorCoordinatorHandler, List<ExceptionHistoryEntry> failureCollection) Transitions into the Canceling state.
void	goToCreatingExecutionGraph() Transitions into the CreatingExecutionGraph state.
void	goToExecuting(ExecutionGraph executionGraph, ExecutionGraphHandler executionGraphHandler, OperatorCoordinatorHandler operatorCoordinatorHandler, List<ExceptionHistoryEntry> failureCollection) Transitions into the Executing state.
void	goToFailing(ExecutionGraph executionGraph, ExecutionGraphHandler executionGraphHandler, OperatorCoordinatorHandler operatorCoordinatorHandler, Throwable failureCause, List<ExceptionHistoryEntry> failureCollection) Transitions into the Failing state.
void	goToFinished(ArchivedExecutionGraph archivedExecutionGraph) Transitions into the Finished state.
void	goToRestarting(ExecutionGraph executionGraph, ExecutionGraphHandler executionGraphHandler, OperatorCoordinatorHandler operatorCoordinatorHandler, java.time.Duration backoffTime, List<ExceptionHistoryEntry> failureCollection) Transitions into the Restarting state.
CompletableFuture<String>	goToStopWithSavepoint(ExecutionGraph executionGraph, ExecutionGraphHandler executionGraphHandler, OperatorCoordinatorHandler operatorCoordinatorHandler, CheckpointScheduling checkpointScheduling, CompletableFuture<String> savepointFuture, List<ExceptionHistoryEntry> failureCollection) Transitions into the StopWithSavepoint state.
void	goToWaitingForResources() Transitions into the WaitingForResources state.
void	handleGlobalFailure(Throwable cause) Handles a global failure.
boolean	hasDesiredResources(ResourceCounter desiredResources) Checks whether we have the desired resources.
boolean	hasSufficientResources() Checks if we currently have sufficient resources for executing the job.
org.apache.flink.runtime.scheduler.adaptive.FailureResult	howToHandleFailure(Throwable failure) Asks how to handle the failure.
boolean	isState(org.apache.flink.runtime.scheduler.adaptive.State expectedState) Checks whether the current state is the expected state.
void	notifyKvStateRegistered(org.apache.flink.api.common.JobID jobId, JobVertexID jobVertexId, KeyGroupRange keyGroupRange, String registrationName, org.apache.flink.queryablestate.KvStateID kvStateId, InetSocketAddress kvStateServerAddress)
void	notifyKvStateUnregistered(org.apache.flink.api.common.JobID jobId, JobVertexID jobVertexId, KeyGroupRange keyGroupRange, String registrationName)
void	onFinished(ArchivedExecutionGraph archivedExecutionGraph) Callback which is called when the execution reaches the Finished state.
void	reportCheckpointMetrics(org.apache.flink.api.common.JobID jobID, ExecutionAttemptID executionAttemptID, long checkpointId, CheckpointMetrics checkpointMetrics)
ExecutionGraphInfo	requestJob()
JobDetails	requestJobDetails()
org.apache.flink.api.common.JobStatus	requestJobStatus()
KvStateLocation	requestKvStateLocation(org.apache.flink.api.common.JobID jobId, String registrationName)
SerializedInputSplit	requestNextInputSplit(JobVertexID vertexID, ExecutionAttemptID executionAttempt)
ExecutionState	requestPartitionState(IntermediateDataSetID intermediateResultId, ResultPartitionID resultPartitionId)
void	runIfState(org.apache.flink.runtime.scheduler.adaptive.State expectedState, Runnable action) Run the given action if the current state equals the expected state.
ScheduledFuture<?>	runIfState(org.apache.flink.runtime.scheduler.adaptive.State expectedState, Runnable action, java.time.Duration delay) Runs the given action after a delay if the state at this time equals the expected state.
void	startScheduling()
CompletableFuture<String>	stopWithSavepoint(String targetDirectory, boolean terminate, org.apache.flink.core.execution.SavepointFormatType formatType)
CompletableFuture<String>	triggerCheckpoint()
CompletableFuture<String>	triggerSavepoint(String targetDirectory, boolean cancelJob, org.apache.flink.core.execution.SavepointFormatType formatType)
org.apache.flink.runtime.scheduler.adaptive.CreatingExecutionGraph.AssignmentResult	tryToAssignSlots(org.apache.flink.runtime.scheduler.adaptive.CreatingExecutionGraph.ExecutionGraphWithVertexParallelism executionGraphWithVertexParallelism) Try to assign slots to the created ExecutionGraph.
void	updateAccumulators(AccumulatorSnapshot accumulatorSnapshot)
boolean	updateTaskExecutionState(TaskExecutionStateTransition taskExecutionState)

从类继承的方法 java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

从接口继承的方法 org.apache.flink.runtime.scheduler.SchedulerNG

updateTaskExecutionState

从接口继承的方法 org.apache.flink.util.AutoCloseableAsync

close

构造器详细资料

AdaptiveScheduler

public AdaptiveScheduler(JobGraph jobGraph,
                         org.apache.flink.configuration.Configuration configuration,
                         DeclarativeSlotPool declarativeSlotPool,
                         SlotAllocator slotAllocator,
                         Executor ioExecutor,
                         ClassLoader userCodeClassLoader,
                         CheckpointsCleaner checkpointsCleaner,
                         CheckpointRecoveryFactory checkpointRecoveryFactory,
                         java.time.Duration initialResourceAllocationTimeout,
                         java.time.Duration resourceStabilizationTimeout,
                         JobManagerJobMetricGroup jobManagerJobMetricGroup,
                         RestartBackoffTimeStrategy restartBackoffTimeStrategy,
                         long initializationTimestamp,
                         org.apache.flink.runtime.concurrent.ComponentMainThreadExecutor mainThreadExecutor,
                         org.apache.flink.runtime.rpc.FatalErrorHandler fatalErrorHandler,
                         JobStatusListener jobStatusListener,
                         ExecutionGraphFactory executionGraphFactory)
                  throws JobExecutionException

抛出:

JobExecutionException

方法详细资料

startScheduling

public void startScheduling()

指定者:

startScheduling 在接口中 SchedulerNG

closeAsync

public CompletableFuture<Void> closeAsync()

指定者:

closeAsync 在接口中 org.apache.flink.util.AutoCloseableAsync

cancel

public void cancel()

指定者:

cancel 在接口中 SchedulerNG

getJobTerminationFuture

public CompletableFuture<org.apache.flink.api.common.JobStatus> getJobTerminationFuture()

指定者:

getJobTerminationFuture 在接口中 SchedulerNG

handleGlobalFailure

public void handleGlobalFailure(Throwable cause)

从接口复制的说明: GlobalFailureHandler

Handles a global failure.

指定者:

handleGlobalFailure 在接口中 GlobalFailureHandler

参数:

cause - A cause that describes the global failure.

updateTaskExecutionState

public boolean updateTaskExecutionState(TaskExecutionStateTransition taskExecutionState)

指定者:

updateTaskExecutionState 在接口中 SchedulerNG

requestNextInputSplit

public SerializedInputSplit requestNextInputSplit(JobVertexID vertexID,
                                                  ExecutionAttemptID executionAttempt)
                                           throws IOException

指定者:

requestNextInputSplit 在接口中 SchedulerNG

抛出:

IOException

requestPartitionState

public ExecutionState requestPartitionState(IntermediateDataSetID intermediateResultId,
                                            ResultPartitionID resultPartitionId)
                                     throws PartitionProducerDisposedException

指定者:

requestPartitionState 在接口中 SchedulerNG

抛出:

PartitionProducerDisposedException

requestJob

public ExecutionGraphInfo requestJob()

指定者:

requestJob 在接口中 SchedulerNG

archiveFailure

public void archiveFailure(RootExceptionHistoryEntry failure)

Archive failure.

requestJobStatus

public org.apache.flink.api.common.JobStatus requestJobStatus()

指定者:

requestJobStatus 在接口中 SchedulerNG

requestJobDetails

public JobDetails requestJobDetails()

指定者:

requestJobDetails 在接口中 SchedulerNG

requestKvStateLocation

public KvStateLocation requestKvStateLocation(org.apache.flink.api.common.JobID jobId,
                                              String registrationName)
                                       throws UnknownKvStateLocation,
                                              FlinkJobNotFoundException

指定者:

requestKvStateLocation 在接口中 SchedulerNG

抛出:

UnknownKvStateLocation

FlinkJobNotFoundException

notifyKvStateRegistered

public void notifyKvStateRegistered(org.apache.flink.api.common.JobID jobId,
                                    JobVertexID jobVertexId,
                                    KeyGroupRange keyGroupRange,
                                    String registrationName,
                                    org.apache.flink.queryablestate.KvStateID kvStateId,
                                    InetSocketAddress kvStateServerAddress)
                             throws FlinkJobNotFoundException

指定者:

notifyKvStateRegistered 在接口中 SchedulerNG

抛出:

FlinkJobNotFoundException

notifyKvStateUnregistered

public void notifyKvStateUnregistered(org.apache.flink.api.common.JobID jobId,
                                      JobVertexID jobVertexId,
                                      KeyGroupRange keyGroupRange,
                                      String registrationName)
                               throws FlinkJobNotFoundException

指定者:

notifyKvStateUnregistered 在接口中 SchedulerNG

抛出:

FlinkJobNotFoundException

updateAccumulators

public void updateAccumulators(AccumulatorSnapshot accumulatorSnapshot)

指定者:

updateAccumulators 在接口中 SchedulerNG

triggerSavepoint

public CompletableFuture<String> triggerSavepoint(@Nullable
                                                  String targetDirectory,
                                                  boolean cancelJob,
                                                  org.apache.flink.core.execution.SavepointFormatType formatType)

指定者:

triggerSavepoint 在接口中 SchedulerNG

triggerCheckpoint

public CompletableFuture<String> triggerCheckpoint()

指定者:

triggerCheckpoint 在接口中 SchedulerNG

acknowledgeCheckpoint

public void acknowledgeCheckpoint(org.apache.flink.api.common.JobID jobID,
                                  ExecutionAttemptID executionAttemptID,
                                  long checkpointId,
                                  CheckpointMetrics checkpointMetrics,
                                  TaskStateSnapshot checkpointState)

指定者:

acknowledgeCheckpoint 在接口中 SchedulerNG

reportCheckpointMetrics

public void reportCheckpointMetrics(org.apache.flink.api.common.JobID jobID,
                                    ExecutionAttemptID executionAttemptID,
                                    long checkpointId,
                                    CheckpointMetrics checkpointMetrics)

指定者:

reportCheckpointMetrics 在接口中 SchedulerNG

declineCheckpoint

public void declineCheckpoint(DeclineCheckpoint decline)

指定者:

declineCheckpoint 在接口中 SchedulerNG

stopWithSavepoint

public CompletableFuture<String> stopWithSavepoint(@Nullable
                                                   String targetDirectory,
                                                   boolean terminate,
                                                   org.apache.flink.core.execution.SavepointFormatType formatType)

指定者:

stopWithSavepoint 在接口中 SchedulerNG

deliverOperatorEventToCoordinator

public void deliverOperatorEventToCoordinator(ExecutionAttemptID taskExecution,
                                              OperatorID operator,
                                              OperatorEvent evt)
                                       throws org.apache.flink.util.FlinkException

从接口复制的说明: SchedulerNG

Delivers the given OperatorEvent to the OperatorCoordinator with the given OperatorID.

Failure semantics: If the task manager sends an event for a non-running task or a non-existing operator coordinator, then respond with an exception to the call. If task and coordinator exist, then we assume that the call from the TaskManager was valid, and any bubbling exception needs to cause a job failure

指定者:

deliverOperatorEventToCoordinator 在接口中 SchedulerNG

抛出:

org.apache.flink.util.FlinkException - Thrown, if the task is not running or no operator/coordinator exists for the given ID.

deliverCoordinationRequestToCoordinator

public CompletableFuture<CoordinationResponse> deliverCoordinationRequestToCoordinator(OperatorID operator,
                                                                                       CoordinationRequest request)
                                                                                throws org.apache.flink.util.FlinkException

从接口复制的说明: SchedulerNG

Delivers a coordination request to the OperatorCoordinator with the given OperatorID and returns the coordinator's response.

指定者:

deliverCoordinationRequestToCoordinator 在接口中 SchedulerNG

返回:

A future containing the response.

抛出:

org.apache.flink.util.FlinkException - Thrown, if the task is not running, or no operator/coordinator exists for the given ID, or the coordinator cannot handle client events.

hasDesiredResources

public boolean hasDesiredResources(ResourceCounter desiredResources)

Checks whether we have the desired resources.

参数:

desiredResources - desiredResources describing the desired resources

返回:

true if we have enough resources; otherwise false

hasSufficientResources

public boolean hasSufficientResources()

Checks if we currently have sufficient resources for executing the job.

返回:

true if we have sufficient resources; otherwise false

getArchivedExecutionGraph

public ArchivedExecutionGraph getArchivedExecutionGraph(org.apache.flink.api.common.JobStatus jobStatus,
                                                        @Nullable
                                                        Throwable cause)

Creates an ArchivedExecutionGraph for the given jobStatus and failure cause.

参数:

jobStatus - jobStatus to create the ArchivedExecutionGraph with

cause - cause represents the failure cause for the ArchivedExecutionGraph; null if there is no failure cause

返回:

the created ArchivedExecutionGraph

goToWaitingForResources

public void goToWaitingForResources()

从接口复制的说明: StateTransitions.ToWaitingForResources

Transitions into the WaitingForResources state.

goToExecuting

public void goToExecuting(ExecutionGraph executionGraph,
                          ExecutionGraphHandler executionGraphHandler,
                          OperatorCoordinatorHandler operatorCoordinatorHandler,
                          List<ExceptionHistoryEntry> failureCollection)

从接口复制的说明: StateTransitions.ToExecuting

Transitions into the Executing state.

参数:

executionGraph - executionGraph to pass to the Executing state

executionGraphHandler - executionGraphHandler to pass to the Executing state

operatorCoordinatorHandler - operatorCoordinatorHandler to pass to the Executing state

failureCollection - collection of failures that are propagated

goToCanceling

public void goToCanceling(ExecutionGraph executionGraph,
                          ExecutionGraphHandler executionGraphHandler,
                          OperatorCoordinatorHandler operatorCoordinatorHandler,
                          List<ExceptionHistoryEntry> failureCollection)

从接口复制的说明: StateTransitions.ToCancelling

Transitions into the Canceling state.

参数:

executionGraph - executionGraph to pass to the Canceling state

executionGraphHandler - executionGraphHandler to pass to the Canceling state

operatorCoordinatorHandler - operatorCoordinatorHandler to pass to the Canceling state

failureCollection - collection of failures that are propagated

goToRestarting

public void goToRestarting(ExecutionGraph executionGraph,
                           ExecutionGraphHandler executionGraphHandler,
                           OperatorCoordinatorHandler operatorCoordinatorHandler,
                           java.time.Duration backoffTime,
                           List<ExceptionHistoryEntry> failureCollection)

从接口复制的说明: StateTransitions.ToRestarting

Transitions into the Restarting state.

参数:

executionGraph - executionGraph to pass to the Restarting state

executionGraphHandler - executionGraphHandler to pass to the Restarting state

operatorCoordinatorHandler - operatorCoordinatorHandler to pas to the Restarting state

backoffTime - backoffTime to wait before transitioning to the Restarting state

failureCollection - collection of failures that are propagated

goToFailing

public void goToFailing(ExecutionGraph executionGraph,
                        ExecutionGraphHandler executionGraphHandler,
                        OperatorCoordinatorHandler operatorCoordinatorHandler,
                        Throwable failureCause,
                        List<ExceptionHistoryEntry> failureCollection)

从接口复制的说明: StateTransitions.ToFailing

Transitions into the Failing state.

参数:

executionGraph - executionGraph to pass to the Failing state

executionGraphHandler - executionGraphHandler to pass to the Failing state

operatorCoordinatorHandler - operatorCoordinatorHandler to pass to the Failing state

failureCause - failureCause describing why the job execution failed

failureCollection - collection of failures that are propagated

goToStopWithSavepoint

public CompletableFuture<String> goToStopWithSavepoint(ExecutionGraph executionGraph,
                                                       ExecutionGraphHandler executionGraphHandler,
                                                       OperatorCoordinatorHandler operatorCoordinatorHandler,
                                                       CheckpointScheduling checkpointScheduling,
                                                       CompletableFuture<String> savepointFuture,
                                                       List<ExceptionHistoryEntry> failureCollection)

从接口复制的说明: StateTransitions.ToStopWithSavepoint

Transitions into the StopWithSavepoint state.

参数:

executionGraph - executionGraph to pass to the StopWithSavepoint state

executionGraphHandler - executionGraphHandler to pass to the StopWithSavepoint state

operatorCoordinatorHandler - operatorCoordinatorHandler to pass to the StopWithSavepoint state

savepointFuture - Future for the savepoint to complete.

failureCollection - collection of failures that are propagated

返回:

Location of the savepoint.

goToFinished

public void goToFinished(ArchivedExecutionGraph archivedExecutionGraph)

从接口复制的说明: StateTransitions.ToFinished

Transitions into the Finished state.

参数:

archivedExecutionGraph - archivedExecutionGraph which is passed to the Finished state

goToCreatingExecutionGraph

public void goToCreatingExecutionGraph()

从接口复制的说明: StateTransitions.ToCreatingExecutionGraph

Transitions into the CreatingExecutionGraph state.

tryToAssignSlots

public org.apache.flink.runtime.scheduler.adaptive.CreatingExecutionGraph.AssignmentResult tryToAssignSlots(org.apache.flink.runtime.scheduler.adaptive.CreatingExecutionGraph.ExecutionGraphWithVertexParallelism executionGraphWithVertexParallelism)

Try to assign slots to the created ExecutionGraph. If it is possible, then this method returns a successful AssignmentResult which contains the assigned ExecutionGraph. If not, then the assignment result is a failure.

参数:

executionGraphWithVertexParallelism - executionGraphWithVertexParallelism to assign slots to resources

返回:

AssignmentResult representing the result of the assignment

canScaleUp

public boolean canScaleUp(ExecutionGraph executionGraph)

Asks if we can scale up the currently executing job.

参数:

executionGraph - executionGraph for making the scaling decision.

返回:

true, if we can scale up

onFinished

public void onFinished(ArchivedExecutionGraph archivedExecutionGraph)

Callback which is called when the execution reaches the Finished state.

参数:

archivedExecutionGraph - archivedExecutionGraph represents the final state of the job execution

howToHandleFailure

public org.apache.flink.runtime.scheduler.adaptive.FailureResult howToHandleFailure(Throwable failure)

Asks how to handle the failure.

参数:

failure - failure describing the failure cause

返回:

FailureResult which describes how to handle the failure

getIOExecutor

public Executor getIOExecutor()

Gets the I/O executor.

返回:

the I/O executor

getMainThreadExecutor

public org.apache.flink.runtime.concurrent.ComponentMainThreadExecutor getMainThreadExecutor()

Gets the main thread executor.

返回:

the main thread executor

isState

public boolean isState(org.apache.flink.runtime.scheduler.adaptive.State expectedState)

Checks whether the current state is the expected state.

参数:

expectedState - expectedState is the expected state

返回:

true if the current state equals the expected state; otherwise false

runIfState

public void runIfState(org.apache.flink.runtime.scheduler.adaptive.State expectedState,
                       Runnable action)

Run the given action if the current state equals the expected state.

参数:

expectedState - expectedState is the expected state

action - action to run if the current state equals the expected state

runIfState

public ScheduledFuture<?> runIfState(org.apache.flink.runtime.scheduler.adaptive.State expectedState,
                                     Runnable action,
                                     java.time.Duration delay)

Runs the given action after a delay if the state at this time equals the expected state.

参数:

expectedState - expectedState describes the required state at the time of running the action

action - action to run if the expected state equals the actual state

delay - delay after which to run the action

返回:

a ScheduledFuture representing pending completion of the task