org.apache.flink.runtime.io.network.partition.hybrid

类 HsResultPartition

java.lang.Object

org.apache.flink.runtime.io.network.partition.ResultPartition

org.apache.flink.runtime.io.network.partition.hybrid.HsResultPartition

所有已实现的接口:

AutoCloseable, AvailabilityProvider, ResultPartitionWriter

public class HsResultPartition
extends ResultPartition

HsResultPartition appends records and events to HsMemoryDataManager, the shuffle data maybe spilled to disk according to the HsSpillingStrategy, and the downstream can consume data from memory or disk.

嵌套类概要

从接口继承的嵌套类/接口 org.apache.flink.runtime.io.AvailabilityProvider

AvailabilityProvider.AvailabilityHelper

字段概要

字段

限定符和类型	字段和说明
static String	DATA_FILE_SUFFIX

从类继承的字段 org.apache.flink.runtime.io.network.partition.ResultPartition

bufferCompressor, bufferPool, LOG, numBuffersOut, numBytesOut, numBytesProduced, numSubpartitions, partitionId, partitionManager, partitionType

从接口继承的字段 org.apache.flink.runtime.io.AvailabilityProvider

AVAILABLE

构造器概要

构造器

构造器和说明

HsResultPartition(String owningTaskName, int partitionIndex, ResultPartitionID partitionId, ResultPartitionType partitionType, int numSubpartitions, int numTargetKeyGroups, BatchShuffleReadBufferPool readBufferPool, ScheduledExecutorService readIOExecutor, ResultPartitionManager partitionManager, String dataFileBashPath, int networkBufferSize, HybridShuffleConfiguration hybridShuffleConfiguration, BufferCompressor bufferCompressor, org.apache.flink.util.function.SupplierWithException<BufferPool,IOException> bufferPoolFactory)

方法概要

限定符和类型	方法和说明
void	abortCheckpoint(long checkpointId, CheckpointException cause) Abort the checkpoint.
void	alignedBarrierTimeout(long checkpointId) Timeout the aligned barrier to unaligned barrier.
void	broadcastEvent(AbstractEvent event, boolean isPriorityEvent) Writes the given AbstractEvent to all channels.
void	broadcastRecord(ByteBuffer record) Writes the given serialized record to all subpartitions.
void	close() Closes the partition writer which releases the allocated resource, for example the buffer pool.
ResultSubpartitionView	createSubpartitionView(int subpartitionId, BufferAvailabilityListener availabilityListener) Returns a reader for the subpartition with the given index.
void	emitRecord(ByteBuffer record, int targetSubpartition) Writes the given serialized record to the target subpartition.
void	finish() Finishes the result partition.
void	flush(int subpartitionIndex) Manually trigger the consumption of data from the given subpartitions.
void	flushAll() Manually trigger the consumption of data from all subpartitions.
int	getNumberOfQueuedBuffers() Returns the total number of queued buffers of all subpartitions.
int	getNumberOfQueuedBuffers(int targetSubpartition) Returns the number of queued buffers of the given target subpartition.
long	getSizeOfQueuedBuffersUnsafe() Returns the total size in bytes of queued buffers of all subpartitions.
void	notifyEndOfData(StopMode mode) Notifies the downstream tasks that this ResultPartitionWriter have emitted all the user records.
protected void	releaseInternal() Releases all produced data including both those stored in memory and persisted on disk.
void	setMetricGroup(TaskIOMetricGroup metrics) Sets the metric group for the ResultPartitionWriter.
protected void	setupInternal() Do the subclass's own setup operation.

从类继承的方法 org.apache.flink.runtime.io.network.partition.ResultPartition

canBeCompressed, checkInProduceState, fail, getAllDataProcessedFuture, getAvailableFuture, getBufferPool, getFailureCause, getNumberOfSubpartitions, getNumTargetKeyGroups, getOwningTaskName, getPartitionId, getPartitionIndex, getPartitionManager, getPartitionType, isFinished, isReleased, onSubpartitionAllDataProcessed, release, release, setMaxOverdraftBuffersPerGate, setup, toString

从类继承的方法 java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

从接口继承的方法 org.apache.flink.runtime.io.AvailabilityProvider

and, isApproximatelyAvailable, isAvailable, or

字段详细资料

DATA_FILE_SUFFIX

public static final String DATA_FILE_SUFFIX

另请参阅:

常量字段值

构造器详细资料

HsResultPartition

public HsResultPartition(String owningTaskName,
                         int partitionIndex,
                         ResultPartitionID partitionId,
                         ResultPartitionType partitionType,
                         int numSubpartitions,
                         int numTargetKeyGroups,
                         BatchShuffleReadBufferPool readBufferPool,
                         ScheduledExecutorService readIOExecutor,
                         ResultPartitionManager partitionManager,
                         String dataFileBashPath,
                         int networkBufferSize,
                         HybridShuffleConfiguration hybridShuffleConfiguration,
                         @Nullable
                         BufferCompressor bufferCompressor,
                         org.apache.flink.util.function.SupplierWithException<BufferPool,IOException> bufferPoolFactory)

方法详细资料

setupInternal

protected void setupInternal()
                      throws IOException

从类复制的说明: ResultPartition

Do the subclass's own setup operation.

指定者:

setupInternal 在类中 ResultPartition

抛出:

IOException

setMetricGroup

public void setMetricGroup(TaskIOMetricGroup metrics)

从接口复制的说明: ResultPartitionWriter

Sets the metric group for the ResultPartitionWriter.

指定者:

setMetricGroup 在接口中 ResultPartitionWriter

覆盖:

setMetricGroup 在类中 ResultPartition

emitRecord

public void emitRecord(ByteBuffer record,
                       int targetSubpartition)
                throws IOException

从接口复制的说明: ResultPartitionWriter

Writes the given serialized record to the target subpartition.

抛出:

IOException

broadcastRecord

public void broadcastRecord(ByteBuffer record)
                     throws IOException

从接口复制的说明: ResultPartitionWriter

Writes the given serialized record to all subpartitions. One can also achieve the same effect by emitting the same record to all subpartitions one by one, however, this method can have better performance for the underlying implementation can do some optimizations, for example coping the given serialized record only once to a shared channel which can be consumed by all subpartitions.

抛出:

IOException

broadcastEvent

public void broadcastEvent(AbstractEvent event,
                           boolean isPriorityEvent)
                    throws IOException

从接口复制的说明: ResultPartitionWriter

Writes the given AbstractEvent to all channels.

抛出:

IOException

createSubpartitionView

public ResultSubpartitionView createSubpartitionView(int subpartitionId,
                                                     BufferAvailabilityListener availabilityListener)
                                              throws IOException

从接口复制的说明: ResultPartitionWriter

Returns a reader for the subpartition with the given index.

抛出:

IOException

alignedBarrierTimeout

public void alignedBarrierTimeout(long checkpointId)
                           throws IOException

从接口复制的说明: ResultPartitionWriter

Timeout the aligned barrier to unaligned barrier.

抛出:

IOException

abortCheckpoint

public void abortCheckpoint(long checkpointId,
                            CheckpointException cause)

从接口复制的说明: ResultPartitionWriter

Abort the checkpoint.

flushAll

public void flushAll()

从接口复制的说明: ResultPartitionWriter

Manually trigger the consumption of data from all subpartitions.

flush

public void flush(int subpartitionIndex)

从接口复制的说明: ResultPartitionWriter

Manually trigger the consumption of data from the given subpartitions.

finish

public void finish()
            throws IOException

从类复制的说明: ResultPartition

Finishes the result partition.

After this operation, it is not possible to add further data to the result partition.

For BLOCKING results, this will trigger the deployment of consuming tasks.

指定者:

finish 在接口中 ResultPartitionWriter

覆盖:

finish 在类中 ResultPartition

抛出:

IOException

close

public void close()

从接口复制的说明: ResultPartitionWriter

Closes the partition writer which releases the allocated resource, for example the buffer pool.

指定者:

close 在接口中 AutoCloseable

指定者:

close 在接口中 ResultPartitionWriter

覆盖:

close 在类中 ResultPartition

releaseInternal

protected void releaseInternal()

从类复制的说明: ResultPartition

Releases all produced data including both those stored in memory and persisted on disk.

指定者:

releaseInternal 在类中 ResultPartition

getNumberOfQueuedBuffers

public int getNumberOfQueuedBuffers()

从类复制的说明: ResultPartition

Returns the total number of queued buffers of all subpartitions.

指定者:

getNumberOfQueuedBuffers 在类中 ResultPartition

getSizeOfQueuedBuffersUnsafe

public long getSizeOfQueuedBuffersUnsafe()

从类复制的说明: ResultPartition

Returns the total size in bytes of queued buffers of all subpartitions.

指定者:

getSizeOfQueuedBuffersUnsafe 在类中 ResultPartition

getNumberOfQueuedBuffers

public int getNumberOfQueuedBuffers(int targetSubpartition)

从类复制的说明: ResultPartition

Returns the number of queued buffers of the given target subpartition.

指定者:

getNumberOfQueuedBuffers 在类中 ResultPartition

notifyEndOfData

public void notifyEndOfData(StopMode mode)
                     throws IOException

从接口复制的说明: ResultPartitionWriter

Notifies the downstream tasks that this ResultPartitionWriter have emitted all the user records.

指定者:

notifyEndOfData 在接口中 ResultPartitionWriter

覆盖:

notifyEndOfData 在类中 ResultPartition

参数:

mode - tells if we should flush all records or not (it is false in case of stop-with-savepoint (--no-drain))

抛出:

IOException