public class PipelinedResultPartition extends BufferWritingResultPartition implements CheckpointedResultPartition, ChannelStateHolder
This result partition implementation is used both in batch and streaming. For streaming it supports low latency transfers (ensure data is sent within x milliseconds) or unconstrained while for batch it transfers only once a buffer is full. Additionally, for streaming use this typically limits the length of the buffer backlog to not have too much data in flight, while for batch we do not constrain this.
The PipelinedResultPartition cannot reconnect once a consumer disconnects (finished or
errored). Once all consumers have disconnected (released the subpartition, notified via the call
onConsumedSubpartition(int)
) then the partition as a whole is disposed and all buffers
are freed.
AvailabilityProvider.AvailabilityHelper
subpartitions
bufferCompressor, bufferPool, LOG, numBuffersOut, numBytesOut, numSubpartitions, partitionId, partitionManager, partitionType
AVAILABLE
Constructor and Description |
---|
PipelinedResultPartition(String owningTaskName,
int partitionIndex,
ResultPartitionID partitionId,
ResultPartitionType partitionType,
ResultSubpartition[] subpartitions,
int numTargetKeyGroups,
ResultPartitionManager partitionManager,
BufferCompressor bufferCompressor,
SupplierWithException<BufferPool,IOException> bufferPoolFactory) |
Modifier and Type | Method and Description |
---|---|
void |
close()
Closes the partition writer which releases the allocated resource, for example the buffer
pool.
|
void |
finishReadRecoveredState(boolean notifyAndBlockOnCompletion) |
void |
flush(int targetSubpartition)
Manually trigger the consumption of data from the given subpartitions.
|
void |
flushAll()
Manually trigger the consumption of data from all subpartitions.
|
CheckpointedResultSubpartition |
getCheckpointedSubpartition(int subpartitionIndex)
Gets the checkpointed subpartition with the given subpartitionIndex.
|
void |
setChannelStateWriter(ChannelStateWriter channelStateWriter)
Injects the
ChannelStateWriter . |
String |
toString() |
broadcastEvent, broadcastRecord, createSubpartitionView, emitRecord, finish, flushAllSubpartitions, flushSubpartition, getAllPartitions, getBackPressuredTimeMsPerSecond, getNumberOfQueuedBuffers, getNumberOfQueuedBuffers, releaseInternal, setMetricGroup, setup
canBeCompressed, checkInProduceState, fail, getAvailableFuture, getBufferPool, getFailureCause, getNumberOfSubpartitions, getNumTargetKeyGroups, getOwningTaskName, getPartitionId, getPartitionIndex, getPartitionManager, getPartitionType, isFinished, isReleased, release, release
clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait
and, isApproximatelyAvailable, isAvailable, or
public PipelinedResultPartition(String owningTaskName, int partitionIndex, ResultPartitionID partitionId, ResultPartitionType partitionType, ResultSubpartition[] subpartitions, int numTargetKeyGroups, ResultPartitionManager partitionManager, @Nullable BufferCompressor bufferCompressor, SupplierWithException<BufferPool,IOException> bufferPoolFactory)
public void setChannelStateWriter(ChannelStateWriter channelStateWriter)
ChannelStateHolder
ChannelStateWriter
. Must only be called once.setChannelStateWriter
in interface ChannelStateHolder
public CheckpointedResultSubpartition getCheckpointedSubpartition(int subpartitionIndex)
CheckpointedResultPartition
getCheckpointedSubpartition
in interface CheckpointedResultPartition
public void flushAll()
ResultPartitionWriter
flushAll
in interface ResultPartitionWriter
public void flush(int targetSubpartition)
ResultPartitionWriter
flush
in interface ResultPartitionWriter
public String toString()
toString
in class ResultPartition
public void finishReadRecoveredState(boolean notifyAndBlockOnCompletion) throws IOException
finishReadRecoveredState
in interface CheckpointedResultPartition
IOException
public void close()
ResultPartitionWriter
close
in interface AutoCloseable
close
in interface ResultPartitionWriter
close
in class BufferWritingResultPartition
Copyright © 2014–2022 The Apache Software Foundation. All rights reserved.