org.apache.flink.runtime.scheduler.adaptivebatch

类 AdaptiveBatchScheduler

java.lang.Object

org.apache.flink.runtime.scheduler.SchedulerBase

org.apache.flink.runtime.scheduler.DefaultScheduler

org.apache.flink.runtime.scheduler.adaptivebatch.AdaptiveBatchScheduler

所有已实现的接口:

AutoCloseable, CheckpointScheduling, GlobalFailureHandler, SchedulerNG, SchedulerOperations, org.apache.flink.util.AutoCloseableAsync

直接已知子类:

SpeculativeScheduler

public class AdaptiveBatchScheduler
extends DefaultScheduler

This scheduler decides the parallelism of JobVertex according to the data volume it consumes. A dynamically built up ExecutionGraph is used for this purpose.

字段概要

从类继承的字段 org.apache.flink.runtime.scheduler.DefaultScheduler

executionDeployer, executionSlotAllocator, log

从类继承的字段 org.apache.flink.runtime.scheduler.SchedulerBase

executionVertexVersioner, inputsLocationsRetriever, jobManagerJobMetricGroup, operatorCoordinatorHandler, stateLocationRetriever

构造器概要

构造器

构造器和说明

AdaptiveBatchScheduler(org.slf4j.Logger log, JobGraph jobGraph, Executor ioExecutor, org.apache.flink.configuration.Configuration jobMasterConfiguration, java.util.function.Consumer<org.apache.flink.runtime.concurrent.ComponentMainThreadExecutor> startUpAction, org.apache.flink.util.concurrent.ScheduledExecutor delayExecutor, ClassLoader userCodeLoader, CheckpointsCleaner checkpointsCleaner, CheckpointRecoveryFactory checkpointRecoveryFactory, JobManagerJobMetricGroup jobManagerJobMetricGroup, SchedulingStrategyFactory schedulingStrategyFactory, FailoverStrategy.Factory failoverStrategyFactory, RestartBackoffTimeStrategy restartBackoffTimeStrategy, ExecutionOperations executionOperations, ExecutionVertexVersioner executionVertexVersioner, ExecutionSlotAllocatorFactory executionSlotAllocatorFactory, long initializationTimestamp, org.apache.flink.runtime.concurrent.ComponentMainThreadExecutor mainThreadExecutor, JobStatusListener jobStatusListener, ExecutionGraphFactory executionGraphFactory, ShuffleMaster<?> shuffleMaster, org.apache.flink.api.common.time.Time rpcTimeout, VertexParallelismDecider vertexParallelismDecider, int defaultMaxParallelism)

方法概要

限定符和类型	方法和说明
static VertexParallelismStore	computeVertexParallelismStoreForDynamicGraph(Iterable<JobVertex> vertices, int defaultMaxParallelism) Compute the VertexParallelismStore for all given vertices in a dynamic graph, which will set defaults and ensure that the returned store contains valid parallelisms, with the configured default max parallelism.
protected void	onTaskFinished(Execution execution)
protected void	startSchedulingInternal()

从类继承的方法 org.apache.flink.runtime.scheduler.DefaultScheduler

allocateSlotsAndDeploy, cancelAllPendingSlotRequestsForVertex, cancelAllPendingSlotRequestsInternal, cancelExecution, createFailureHandlingResultSnapshot, getNumberOfRestarts, handleGlobalFailure, handleTaskFailure, onTaskFailed, recordTaskFailure

从类继承的方法 org.apache.flink.runtime.scheduler.SchedulerBase

acknowledgeCheckpoint, archiveFromFailureHandlingResult, archiveGlobalFailure, cancel, closeAsync, computeVertexParallelismStore, computeVertexParallelismStore, computeVertexParallelismStore, computeVertexParallelismStore, declineCheckpoint, deliverCoordinationRequestToCoordinator, deliverOperatorEventToCoordinator, failJob, getDefaultMaxParallelism, getExceptionHistory, getExecutionGraph, getExecutionJobVertex, getExecutionVertex, getJobGraph, getJobTerminationFuture, getMainThreadExecutor, getResultPartitionAvailabilityChecker, getSchedulingTopology, notifyKvStateRegistered, notifyKvStateUnregistered, registerJobMetrics, reportCheckpointMetrics, requestJob, requestJobDetails, requestJobStatus, requestKvStateLocation, requestNextInputSplit, requestPartitionState, resetForNewExecution, resetForNewExecutions, restoreState, setGlobalFailureCause, startCheckpointScheduler, startScheduling, stopCheckpointScheduler, stopWithSavepoint, transitionExecutionGraphState, transitionToRunning, transitionToScheduled, triggerCheckpoint, triggerSavepoint, updateAccumulators, updateTaskExecutionState

从类继承的方法 java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

从接口继承的方法 org.apache.flink.runtime.scheduler.SchedulerNG

updateTaskExecutionState

从接口继承的方法 org.apache.flink.util.AutoCloseableAsync

close

构造器详细资料

AdaptiveBatchScheduler

public AdaptiveBatchScheduler(org.slf4j.Logger log,
                              JobGraph jobGraph,
                              Executor ioExecutor,
                              org.apache.flink.configuration.Configuration jobMasterConfiguration,
                              java.util.function.Consumer<org.apache.flink.runtime.concurrent.ComponentMainThreadExecutor> startUpAction,
                              org.apache.flink.util.concurrent.ScheduledExecutor delayExecutor,
                              ClassLoader userCodeLoader,
                              CheckpointsCleaner checkpointsCleaner,
                              CheckpointRecoveryFactory checkpointRecoveryFactory,
                              JobManagerJobMetricGroup jobManagerJobMetricGroup,
                              SchedulingStrategyFactory schedulingStrategyFactory,
                              FailoverStrategy.Factory failoverStrategyFactory,
                              RestartBackoffTimeStrategy restartBackoffTimeStrategy,
                              ExecutionOperations executionOperations,
                              ExecutionVertexVersioner executionVertexVersioner,
                              ExecutionSlotAllocatorFactory executionSlotAllocatorFactory,
                              long initializationTimestamp,
                              org.apache.flink.runtime.concurrent.ComponentMainThreadExecutor mainThreadExecutor,
                              JobStatusListener jobStatusListener,
                              ExecutionGraphFactory executionGraphFactory,
                              ShuffleMaster<?> shuffleMaster,
                              org.apache.flink.api.common.time.Time rpcTimeout,
                              VertexParallelismDecider vertexParallelismDecider,
                              int defaultMaxParallelism)
                       throws Exception

抛出:

Exception

方法详细资料

startSchedulingInternal

protected void startSchedulingInternal()

覆盖:

startSchedulingInternal 在类中 DefaultScheduler

onTaskFinished

protected void onTaskFinished(Execution execution)

覆盖:

onTaskFinished 在类中 DefaultScheduler

computeVertexParallelismStoreForDynamicGraph

@VisibleForTesting
public static VertexParallelismStore computeVertexParallelismStoreForDynamicGraph(Iterable<JobVertex> vertices,
                                                                                                     int defaultMaxParallelism)

Compute the VertexParallelismStore for all given vertices in a dynamic graph, which will set defaults and ensure that the returned store contains valid parallelisms, with the configured default max parallelism.

参数:

vertices - the vertices to compute parallelism for

defaultMaxParallelism - the global default max parallelism

返回:

the computed parallelism store