public class PipelinedSubpartition extends ResultSubpartition implements CheckpointedResultSubpartition, ChannelStateHolder
Whenever ResultSubpartition.add(BufferConsumer)
adds a finished BufferConsumer
or a second BufferConsumer
(in which case we will assume the first one finished), we will
notify
a read view created via ResultSubpartition.createReadView(BufferAvailabilityListener)
of new data availability. Except
by calling flush()
explicitly, we always only notify when the first finished buffer
turns up and then, the reader has to drain the buffers via pollBuffer()
until its return
value shows no more buffers being available. This results in a buffer queue which is either empty
or has an unfinished BufferConsumer
left from which the notifications will eventually
start again.
Explicit calls to flush()
will force this notification
for any BufferConsumer
present in the queue.
ResultSubpartition.BufferAndBacklog
parent, subpartitionInfo
Modifier and Type | Method and Description |
---|---|
boolean |
add(BufferConsumer bufferConsumer,
int partialRecordLength)
Adds the given buffer.
|
PipelinedSubpartitionView |
createReadView(BufferAvailabilityListener availabilityListener) |
void |
finish() |
void |
finishReadRecoveredState(boolean notifyAndBlockOnCompletion) |
void |
flush() |
int |
getBuffersInBacklog()
Gets the number of non-event buffers in this subpartition.
|
protected long |
getTotalNumberOfBuffers()
Gets the total numbers of buffers (data buffers plus events).
|
protected long |
getTotalNumberOfBytes() |
boolean |
isAvailable(int numCreditsAvailable) |
boolean |
isReleased() |
void |
release() |
BufferBuilder |
requestBufferBuilderBlocking() |
void |
setChannelStateWriter(ChannelStateWriter channelStateWriter)
Injects the
ChannelStateWriter . |
String |
toString() |
int |
unsynchronizedGetNumberOfQueuedBuffers()
Makes a best effort to get the current size of the queue.
|
add, getSubPartitionIndex, getSubpartitionInfo, onConsumedSubpartition
clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait
getSubpartitionInfo
public void setChannelStateWriter(ChannelStateWriter channelStateWriter)
ChannelStateHolder
ChannelStateWriter
. Must only be called once.setChannelStateWriter
in interface ChannelStateHolder
public boolean add(BufferConsumer bufferConsumer, int partialRecordLength)
ResultSubpartition
The request may be executed synchronously, or asynchronously, depending on the implementation.
IMPORTANT: Before adding new BufferConsumer
previously added must
be in finished state. Because of the performance reasons, this is only enforced during the
data reading. Priority events can be added while the previous buffer consumer is still open,
in which case the open buffer consumer is overtaken.
add
in interface CheckpointedResultSubpartition
add
in class ResultSubpartition
bufferConsumer
- the buffer to add (transferring ownership to this writer)partialRecordLength
- the length of bytes to skip in order to start with a complete
record, from position index 0 of the underlying .public void finishReadRecoveredState(boolean notifyAndBlockOnCompletion) throws IOException
finishReadRecoveredState
in interface CheckpointedResultSubpartition
IOException
public void finish() throws IOException
finish
in class ResultSubpartition
IOException
public void release()
release
in class ResultSubpartition
public boolean isReleased()
isReleased
in class ResultSubpartition
public PipelinedSubpartitionView createReadView(BufferAvailabilityListener availabilityListener)
createReadView
in class ResultSubpartition
public boolean isAvailable(int numCreditsAvailable)
public int unsynchronizedGetNumberOfQueuedBuffers()
ResultSubpartition
unsynchronizedGetNumberOfQueuedBuffers
in class ResultSubpartition
public void flush()
flush
in class ResultSubpartition
protected long getTotalNumberOfBuffers()
ResultSubpartition
getTotalNumberOfBuffers
in class ResultSubpartition
protected long getTotalNumberOfBytes()
getTotalNumberOfBytes
in class ResultSubpartition
@VisibleForTesting public int getBuffersInBacklog()
Beware: This method should only be used in tests in non-concurrent access scenarios since it does not make any concurrency guarantees.
public BufferBuilder requestBufferBuilderBlocking() throws InterruptedException
requestBufferBuilderBlocking
in interface CheckpointedResultSubpartition
InterruptedException
Copyright © 2014–2022 The Apache Software Foundation. All rights reserved.