public class ConsumableNotifyingResultPartitionWriterDecorator extends Object implements ResultPartitionWriter
Before a consuming task can request the result, it has to be deployed. The time of deployment depends on the PIPELINED vs. BLOCKING characteristic of the result partition. With pipelined results, receivers are deployed as soon as the first buffer is added to the result partition. With blocking results on the other hand, receivers are deployed after the partition is finished.
Constructor and Description |
---|
ConsumableNotifyingResultPartitionWriterDecorator(TaskActions taskActions,
JobID jobId,
ResultPartitionWriter partitionWriter,
ResultPartitionConsumableNotifier partitionConsumableNotifier) |
Modifier and Type | Method and Description |
---|---|
boolean |
addBufferConsumer(BufferConsumer bufferConsumer,
int subpartitionIndex)
Adds the bufferConsumer to the subpartition with the given index.
|
void |
close() |
static ResultPartitionWriter[] |
decorate(Collection<ResultPartitionDeploymentDescriptor> descs,
ResultPartitionWriter[] partitionWriters,
TaskActions taskActions,
JobID jobId,
ResultPartitionConsumableNotifier notifier) |
void |
fail(Throwable throwable)
Fail the production of the partition.
|
void |
finish()
Successfully finish the production of the partition.
|
void |
flush(int subpartitionIndex)
Manually trigger consumption from enqueued
BufferConsumers in one specified subpartition. |
void |
flushAll()
Manually trigger consumption from enqueued
BufferConsumers in all subpartitions. |
BufferBuilder |
getBufferBuilder()
Requests a
BufferBuilder from this partition for writing data. |
int |
getNumberOfSubpartitions() |
int |
getNumTargetKeyGroups() |
ResultPartitionID |
getPartitionId() |
void |
setup()
Setup partition, potentially heavy-weight, blocking operation comparing to just creation.
|
public ConsumableNotifyingResultPartitionWriterDecorator(TaskActions taskActions, JobID jobId, ResultPartitionWriter partitionWriter, ResultPartitionConsumableNotifier partitionConsumableNotifier)
public BufferBuilder getBufferBuilder() throws IOException, InterruptedException
ResultPartitionWriter
BufferBuilder
from this partition for writing data.getBufferBuilder
in interface ResultPartitionWriter
IOException
InterruptedException
public ResultPartitionID getPartitionId()
getPartitionId
in interface ResultPartitionWriter
public int getNumberOfSubpartitions()
getNumberOfSubpartitions
in interface ResultPartitionWriter
public int getNumTargetKeyGroups()
getNumTargetKeyGroups
in interface ResultPartitionWriter
public void setup() throws IOException
ResultPartitionWriter
setup
in interface ResultPartitionWriter
IOException
public boolean addBufferConsumer(BufferConsumer bufferConsumer, int subpartitionIndex) throws IOException
ResultPartitionWriter
This method takes the ownership of the passed bufferConsumer
and thus is responsible for releasing
it's resources.
To avoid problems with data re-ordering, before adding new BufferConsumer
the previously added one
the given subpartitionIndex
must be marked as BufferConsumer.isFinished()
.
addBufferConsumer
in interface ResultPartitionWriter
IOException
public void flushAll()
ResultPartitionWriter
BufferConsumers
in all subpartitions.flushAll
in interface ResultPartitionWriter
public void flush(int subpartitionIndex)
ResultPartitionWriter
BufferConsumers
in one specified subpartition.flush
in interface ResultPartitionWriter
public void finish() throws IOException
ResultPartitionWriter
Closing of partition is still needed afterwards.
finish
in interface ResultPartitionWriter
IOException
public void fail(Throwable throwable)
ResultPartitionWriter
This method propagates non-null
failure causes to consumers on a best-effort basis. This call also
leads to the release of all resources associated with the partition. Closing of the partition is still needed
afterwards if it has not been done before.
fail
in interface ResultPartitionWriter
throwable
- failure causepublic void close() throws Exception
close
in interface AutoCloseable
Exception
public static ResultPartitionWriter[] decorate(Collection<ResultPartitionDeploymentDescriptor> descs, ResultPartitionWriter[] partitionWriters, TaskActions taskActions, JobID jobId, ResultPartitionConsumableNotifier notifier)
Copyright © 2014–2020 The Apache Software Foundation. All rights reserved.