org.apache.flink.streaming.runtime.partitioner

类 ForwardForUnspecifiedPartitioner<T>

java.lang.Object

org.apache.flink.streaming.runtime.partitioner.StreamPartitioner<T>

org.apache.flink.streaming.runtime.partitioner.ForwardPartitioner<T>

org.apache.flink.streaming.runtime.partitioner.ForwardForUnspecifiedPartitioner<T>

类型参数:

T - Type of the elements in the Stream

所有已实现的接口:

Serializable, org.apache.flink.runtime.io.network.api.writer.ChannelSelector<org.apache.flink.runtime.plugable.SerializationDelegate<StreamRecord<T>>>

@Internal
public class ForwardForUnspecifiedPartitioner<T>
extends ForwardPartitioner<T>

When the parallelism of both upstream and downstream is ExecutionConfig.PARALLELISM_DEFAULT and the edge's partitioner is not specified (partitioner==null), the edge's partitioner will be set to FORWARD by default(See StreamGraph.createActualEdge(java.lang.Integer, java.lang.Integer, int, org.apache.flink.streaming.runtime.partitioner.StreamPartitioner<?>, org.apache.flink.util.OutputTag, org.apache.flink.streaming.api.transformations.StreamExchangeMode, org.apache.flink.runtime.jobgraph.IntermediateDataSetID) method for details). When using the AdaptiveBatchScheduler, this will result in the parallelism of many job vertices is not calculated based on the amount of data but has to align with the parallelism of their upstream vertices due to forward edges, which is contrary to the original intention of the AdaptiveBatchScheduler.

To solve it, we introduce the ForwardForUnspecifiedPartitioner. This partitioner will be set for unspecified edges(partitioner==null), and then the runtime framework will change it to FORWARD/RESCALE after the operator chain creation:

1. Convert to ForwardPartitioner if the partitioner is intra-chain.

2. Convert to RescalePartitioner if the partitioner is inter-chain.

This partitioner should only be used when using AdaptiveBatchScheduler.

另请参阅:

序列化表格

字段概要

从类继承的字段 org.apache.flink.streaming.runtime.partitioner.StreamPartitioner

numberOfChannels

构造器概要

构造器

构造器和说明
ForwardForUnspecifiedPartitioner()

方法概要

限定符和类型	方法和说明
StreamPartitioner<T>	copy()
org.apache.flink.runtime.io.network.api.writer.SubtaskStateMapper	getDownstreamSubtaskStateMapper() Defines the behavior of this partitioner, when downstream rescaled during recovery of in-flight data.
int	selectChannel(org.apache.flink.runtime.plugable.SerializationDelegate<StreamRecord<T>> record)

从类继承的方法 org.apache.flink.streaming.runtime.partitioner.ForwardPartitioner

getUpstreamSubtaskStateMapper, isPointwise, toString

从类继承的方法 org.apache.flink.streaming.runtime.partitioner.StreamPartitioner

equals, hashCode, isBroadcast, setup

从类继承的方法 java.lang.Object

clone, finalize, getClass, notify, notifyAll, wait, wait, wait

构造器详细资料

ForwardForUnspecifiedPartitioner

public ForwardForUnspecifiedPartitioner()

方法详细资料

copy

public StreamPartitioner<T> copy()

覆盖:

copy 在类中 ForwardPartitioner<T>

getDownstreamSubtaskStateMapper

public org.apache.flink.runtime.io.network.api.writer.SubtaskStateMapper getDownstreamSubtaskStateMapper()

从类复制的说明: StreamPartitioner

Defines the behavior of this partitioner, when downstream rescaled during recovery of in-flight data.

覆盖:

getDownstreamSubtaskStateMapper 在类中 ForwardPartitioner<T>

selectChannel

public int selectChannel(org.apache.flink.runtime.plugable.SerializationDelegate<StreamRecord<T>> record)

指定者:

selectChannel 在接口中 org.apache.flink.runtime.io.network.api.writer.ChannelSelector<org.apache.flink.runtime.plugable.SerializationDelegate<StreamRecord<T>>>

覆盖:

selectChannel 在类中 ForwardPartitioner<T>