org.apache.flink.runtime.highavailability

类 AbstractHaServices

java.lang.Object

org.apache.flink.runtime.highavailability.AbstractHaServices

所有已实现的接口:

AutoCloseable, GloballyCleanableResource, ClientHighAvailabilityServices, HighAvailabilityServices

直接已知子类:

AbstractZooKeeperHaServices

public abstract class AbstractHaServices
extends Object
implements HighAvailabilityServices

Abstract high availability services based on distributed system(e.g. Zookeeper, Kubernetes). It will help with creating all the leader election/retrieval services and the cleanup. Please return a proper leader name int the implementation of getLeaderPathForResourceManager(), getLeaderPathForDispatcher(), getLeaderPathForJobManager(org.apache.flink.api.common.JobID), getLeaderPathForRestServer(). The returned leader name is the ConfigMap name in Kubernetes and child path in Zookeeper.

close() and closeAndCleanupAllData() should be implemented to destroy the resources.

The abstract class is also responsible for determining which component service should be reused. For example, jobResultStore is created once and could be reused many times.

字段概要

字段

限定符和类型	字段和说明
protected org.apache.flink.configuration.Configuration	configuration The runtime configuration.
protected Executor	ioExecutor The executor to run external IO operations on.
protected org.slf4j.Logger	logger

从接口继承的字段 org.apache.flink.runtime.highavailability.HighAvailabilityServices

DEFAULT_JOB_ID, DEFAULT_LEADER_ID

构造器概要

构造器

限定符	构造器和说明
protected	AbstractHaServices(org.apache.flink.configuration.Configuration config, Executor ioExecutor, BlobStoreService blobStoreService, JobResultStore jobResultStore)

方法概要

限定符和类型	方法和说明
void	close() Closes the high availability services, releasing all resources.
void	closeAndCleanupAllData() Closes the high availability services (releasing all resources) and deletes all data stored by these services in external stores.
BlobStore	createBlobStore() Creates the BLOB store in which BLOBs are stored in a highly-available fashion.
protected abstract CheckpointRecoveryFactory	createCheckpointRecoveryFactory() Create the checkpoint recovery factory for the job manager.
protected abstract JobGraphStore	createJobGraphStore() Create the submitted job graph store for the job manager.
protected abstract LeaderElectionService	createLeaderElectionService(String leaderName) Create leader election service with specified leaderName.
protected abstract LeaderRetrievalService	createLeaderRetrievalService(String leaderName) Create leader retrieval service with specified leaderName.
CheckpointRecoveryFactory	getCheckpointRecoveryFactory() Gets the checkpoint recovery factory for the job manager.
LeaderElectionService	getClusterRestEndpointLeaderElectionService() Gets the leader election service for the cluster's rest endpoint.
LeaderRetrievalService	getClusterRestEndpointLeaderRetriever() Get the leader retriever for the cluster's rest endpoint.
LeaderElectionService	getDispatcherLeaderElectionService() Gets the leader election service for the cluster's dispatcher.
LeaderRetrievalService	getDispatcherLeaderRetriever() Gets the leader retriever for the dispatcher.
JobGraphStore	getJobGraphStore() Gets the submitted job graph store for the job manager.
LeaderElectionService	getJobManagerLeaderElectionService(org.apache.flink.api.common.JobID jobID) Gets the leader election service for the given job.
LeaderRetrievalService	getJobManagerLeaderRetriever(org.apache.flink.api.common.JobID jobID) Gets the leader retriever for the job JobMaster which is responsible for the given job.
LeaderRetrievalService	getJobManagerLeaderRetriever(org.apache.flink.api.common.JobID jobID, String defaultJobManagerAddress) Gets the leader retriever for the job JobMaster which is responsible for the given job.
JobResultStore	getJobResultStore() Gets the store that holds information about the state of finished jobs.
protected abstract String	getLeaderPathForDispatcher() Get the leader path for Dispatcher.
protected abstract String	getLeaderPathForJobManager(org.apache.flink.api.common.JobID jobID) Get the leader path for specific JobManager.
protected abstract String	getLeaderPathForResourceManager() Get the leader path for ResourceManager.
protected abstract String	getLeaderPathForRestServer() Get the leader path for RestServer.
LeaderElectionService	getResourceManagerLeaderElectionService() Gets the leader election service for the cluster's resource manager.
LeaderRetrievalService	getResourceManagerLeaderRetriever() Gets the leader retriever for the cluster's resource manager.
CompletableFuture<Void>	globalCleanupAsync(org.apache.flink.api.common.JobID jobID, Executor executor) globalCleanupAsync is expected to be called from the main thread.
protected abstract void	internalCleanup() Clean up the meta data in the distributed system(e.g.
protected abstract void	internalCleanupJobData(org.apache.flink.api.common.JobID jobID) Clean up the meta data in the distributed system(e.g.
protected abstract void	internalClose() Closes the components which is used for external operations(e.g.

从类继承的方法 java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

从接口继承的方法 org.apache.flink.runtime.highavailability.HighAvailabilityServices

getWebMonitorLeaderElectionService, getWebMonitorLeaderRetriever

字段详细资料

logger

protected final org.slf4j.Logger logger

ioExecutor

protected final Executor ioExecutor

The executor to run external IO operations on.

configuration

protected final org.apache.flink.configuration.Configuration configuration

The runtime configuration.

构造器详细资料

AbstractHaServices

protected AbstractHaServices(org.apache.flink.configuration.Configuration config,
                             Executor ioExecutor,
                             BlobStoreService blobStoreService,
                             JobResultStore jobResultStore)

方法详细资料

getResourceManagerLeaderRetriever

public LeaderRetrievalService getResourceManagerLeaderRetriever()

从接口复制的说明: HighAvailabilityServices

Gets the leader retriever for the cluster's resource manager.

指定者:

getResourceManagerLeaderRetriever 在接口中 HighAvailabilityServices

getDispatcherLeaderRetriever

public LeaderRetrievalService getDispatcherLeaderRetriever()

从接口复制的说明: HighAvailabilityServices

Gets the leader retriever for the dispatcher. This leader retrieval service is not always accessible.

指定者:

getDispatcherLeaderRetriever 在接口中 HighAvailabilityServices

getJobManagerLeaderRetriever

public LeaderRetrievalService getJobManagerLeaderRetriever(org.apache.flink.api.common.JobID jobID)

从接口复制的说明: HighAvailabilityServices

Gets the leader retriever for the job JobMaster which is responsible for the given job.

指定者:

getJobManagerLeaderRetriever 在接口中 HighAvailabilityServices

参数:

jobID - The identifier of the job.

返回:

Leader retrieval service to retrieve the job manager for the given job

getJobManagerLeaderRetriever

public LeaderRetrievalService getJobManagerLeaderRetriever(org.apache.flink.api.common.JobID jobID,
                                                           String defaultJobManagerAddress)

从接口复制的说明: HighAvailabilityServices

Gets the leader retriever for the job JobMaster which is responsible for the given job.

指定者:

getJobManagerLeaderRetriever 在接口中 HighAvailabilityServices

参数:

jobID - The identifier of the job.

defaultJobManagerAddress - JobManager address which will be returned by a static leader retrieval service.

返回:

Leader retrieval service to retrieve the job manager for the given job

getClusterRestEndpointLeaderRetriever

public LeaderRetrievalService getClusterRestEndpointLeaderRetriever()

从接口复制的说明: ClientHighAvailabilityServices

Get the leader retriever for the cluster's rest endpoint.

指定者:

getClusterRestEndpointLeaderRetriever 在接口中 ClientHighAvailabilityServices

指定者:

getClusterRestEndpointLeaderRetriever 在接口中 HighAvailabilityServices

返回:

the leader retriever for cluster's rest endpoint.

getResourceManagerLeaderElectionService

public LeaderElectionService getResourceManagerLeaderElectionService()

从接口复制的说明: HighAvailabilityServices

Gets the leader election service for the cluster's resource manager.

指定者:

getResourceManagerLeaderElectionService 在接口中 HighAvailabilityServices

返回:

Leader election service for the resource manager leader election

getDispatcherLeaderElectionService

public LeaderElectionService getDispatcherLeaderElectionService()

从接口复制的说明: HighAvailabilityServices

Gets the leader election service for the cluster's dispatcher.

指定者:

getDispatcherLeaderElectionService 在接口中 HighAvailabilityServices

返回:

Leader election service for the dispatcher leader election

getJobManagerLeaderElectionService

public LeaderElectionService getJobManagerLeaderElectionService(org.apache.flink.api.common.JobID jobID)

从接口复制的说明: HighAvailabilityServices

Gets the leader election service for the given job.

指定者:

getJobManagerLeaderElectionService 在接口中 HighAvailabilityServices

参数:

jobID - The identifier of the job running the election.

返回:

Leader election service for the job manager leader election

getClusterRestEndpointLeaderElectionService

public LeaderElectionService getClusterRestEndpointLeaderElectionService()

从接口复制的说明: HighAvailabilityServices

Gets the leader election service for the cluster's rest endpoint.

指定者:

getClusterRestEndpointLeaderElectionService 在接口中 HighAvailabilityServices

返回:

the leader election service used by the cluster's rest endpoint

getCheckpointRecoveryFactory

public CheckpointRecoveryFactory getCheckpointRecoveryFactory()
                                                       throws Exception

从接口复制的说明: HighAvailabilityServices

Gets the checkpoint recovery factory for the job manager.

指定者:

getCheckpointRecoveryFactory 在接口中 HighAvailabilityServices

返回:

Checkpoint recovery factory

抛出:

Exception

getJobGraphStore

public JobGraphStore getJobGraphStore()
                               throws Exception

从接口复制的说明: HighAvailabilityServices

Gets the submitted job graph store for the job manager.

指定者:

getJobGraphStore 在接口中 HighAvailabilityServices

返回:

Submitted job graph store

抛出:

Exception - if the submitted job graph store could not be created

getJobResultStore

public JobResultStore getJobResultStore()
                                 throws Exception

从接口复制的说明: HighAvailabilityServices

Gets the store that holds information about the state of finished jobs.

指定者:

getJobResultStore 在接口中 HighAvailabilityServices

返回:

Store of finished job results

抛出:

Exception - if job result store could not be created

createBlobStore

public BlobStore createBlobStore()

从接口复制的说明: HighAvailabilityServices

Creates the BLOB store in which BLOBs are stored in a highly-available fashion.

指定者:

createBlobStore 在接口中 HighAvailabilityServices

返回:

Blob store

close

public void close()
           throws Exception

从接口复制的说明: HighAvailabilityServices

Closes the high availability services, releasing all resources.

This method does not delete or clean up any data stored in external stores (file systems, ZooKeeper, etc). Another instance of the high availability services will be able to recover the job.

If an exception occurs during closing services, this method will attempt to continue closing other services and report exceptions only after all services have been attempted to be closed.

指定者:

close 在接口中 AutoCloseable

指定者:

close 在接口中 HighAvailabilityServices

抛出:

Exception - Thrown, if an exception occurred while closing these services.

closeAndCleanupAllData

public void closeAndCleanupAllData()
                            throws Exception

从接口复制的说明: HighAvailabilityServices

Closes the high availability services (releasing all resources) and deletes all data stored by these services in external stores.

After this method was called, the any job or session that was managed by these high availability services will be unrecoverable.

If an exception occurs during cleanup, this method will attempt to continue the cleanup and report exceptions only after all cleanup steps have been attempted.

指定者:

closeAndCleanupAllData 在接口中 HighAvailabilityServices

抛出:

Exception - Thrown, if an exception occurred while closing these services or cleaning up data stored by them.

globalCleanupAsync

public CompletableFuture<Void> globalCleanupAsync(org.apache.flink.api.common.JobID jobID,
                                                  Executor executor)

从接口复制的说明: GloballyCleanableResource

globalCleanupAsync is expected to be called from the main thread. Heavy IO tasks should be outsourced into the passed cleanupExecutor. Thread-safety must be ensured.

指定者:

globalCleanupAsync 在接口中 GloballyCleanableResource

指定者:

globalCleanupAsync 在接口中 HighAvailabilityServices

参数:

jobID - The JobID of the job for which the local data should be cleaned up.

executor - The fallback executor for IO-heavy operations.

返回:

The cleanup result future.

createLeaderElectionService

protected abstract LeaderElectionService createLeaderElectionService(String leaderName)

Create leader election service with specified leaderName.

参数:

leaderName - ConfigMap name in Kubernetes or child node path in Zookeeper.

返回:

Return LeaderElectionService using Zookeeper or Kubernetes.

createLeaderRetrievalService

protected abstract LeaderRetrievalService createLeaderRetrievalService(String leaderName)

Create leader retrieval service with specified leaderName.

参数:

leaderName - ConfigMap name in Kubernetes or child node path in Zookeeper.

返回:

Return LeaderRetrievalService using Zookeeper or Kubernetes.

createCheckpointRecoveryFactory

protected abstract CheckpointRecoveryFactory createCheckpointRecoveryFactory()
                                                                      throws Exception

Create the checkpoint recovery factory for the job manager.

返回:

Checkpoint recovery factory

抛出:

Exception

createJobGraphStore

protected abstract JobGraphStore createJobGraphStore()
                                              throws Exception

Create the submitted job graph store for the job manager.

返回:

Submitted job graph store

抛出:

Exception - if the submitted job graph store could not be created

internalClose

protected abstract void internalClose()
                               throws Exception

Closes the components which is used for external operations(e.g. Zookeeper Client, Kubernetes Client).

抛出:

Exception - if the close operation failed

internalCleanup

protected abstract void internalCleanup()
                                 throws Exception

Clean up the meta data in the distributed system(e.g. Zookeeper, Kubernetes ConfigMap).

If an exception occurs during internal cleanup, we will continue the cleanup in closeAndCleanupAllData() and report exceptions only after all cleanup steps have been attempted.

抛出:

Exception - when do the cleanup operation on external storage.

internalCleanupJobData

protected abstract void internalCleanupJobData(org.apache.flink.api.common.JobID jobID)
                                        throws Exception

Clean up the meta data in the distributed system(e.g. Zookeeper, Kubernetes ConfigMap) for the specified Job. Method implementations need to be thread-safe.

参数:

jobID - The identifier of the job to cleanup.

抛出:

Exception - when do the cleanup operation on external storage.

getLeaderPathForResourceManager

protected abstract String getLeaderPathForResourceManager()

Get the leader path for ResourceManager.

返回:

Return the ResourceManager leader name. It is ConfigMap name in Kubernetes or child node path in Zookeeper.

getLeaderPathForDispatcher

protected abstract String getLeaderPathForDispatcher()

Get the leader path for Dispatcher.

返回:

Return the Dispatcher leader name. It is ConfigMap name in Kubernetes or child node path in Zookeeper.

getLeaderPathForJobManager

protected abstract String getLeaderPathForJobManager(org.apache.flink.api.common.JobID jobID)

Get the leader path for specific JobManager.

参数:

jobID - job id

返回:

Return the JobManager leader name for specified job id. It is ConfigMap name in Kubernetes or child node path in Zookeeper.

getLeaderPathForRestServer

protected abstract String getLeaderPathForRestServer()

Get the leader path for RestServer.

返回:

Return the RestServer leader name. It is ConfigMap name in Kubernetes or child node path in Zookeeper.