T
- the type of the record that can be emitted with this record writerpublic final class BroadcastRecordWriter<T extends IOReadableWritable> extends RecordWriter<T>
The BroadcastRecordWriter extends the RecordWriter
and maintain a single BufferBuilder
for all the channels. Then the serialization results need be copied only once to
this buffer which would be shared for all the channels in a more efficient way.
AvailabilityProvider.AvailabilityHelper
DEFAULT_OUTPUT_FLUSH_THREAD_NAME, idleTimeMsPerSecond, numberOfChannels, rng, serializer
AVAILABLE
Modifier and Type | Method and Description |
---|---|
void |
broadcastEmit(T record)
This is used to broadcast streaming Watermarks in-band with records.
|
void |
clearBuffers()
Closes the
BufferBuilder s for all the channels. |
void |
closeBufferBuilder(int targetChannel)
Marks the current
BufferBuilder as finished and releases the resources for the target
channel. |
void |
emit(T record)
This is used to send regular records.
|
void |
emptyCurrentBufferBuilder(int targetChannel)
Marks the current
BufferBuilder as empty for the target channel. |
void |
flushTargetPartition(int targetChannel)
The flush could be triggered by
randomEmit(IOReadableWritable) , emit(IOReadableWritable) or broadcastEmit(IOReadableWritable) . |
BufferBuilder |
getBufferBuilder(int targetChannel)
The
BufferBuilder may already exist if not filled up last time, otherwise we need
request a new one for this target channel. |
void |
randomEmit(T record)
This is used to send LatencyMarks to a random target channel.
|
BufferBuilder |
requestNewBufferBuilder(int targetChannel)
The request could be from broadcast or non-broadcast modes like
randomEmit(IOReadableWritable) . |
void |
tryFinishCurrentBufferBuilder(int targetChannel)
Marks the current
BufferBuilder as finished if present and clears the state for next
one. |
addBufferConsumer, broadcastEvent, broadcastEvent, checkErroneous, close, copyFromSerializerToTargetChannel, emit, finishBufferBuilder, flushAll, getAvailableFuture, getIdleTimeMsPerSecond, setMetricGroup
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
isApproximatelyAvailable, isAvailable
public void emit(T record) throws IOException, InterruptedException
RecordWriter
emit
in class RecordWriter<T extends IOReadableWritable>
IOException
InterruptedException
public void randomEmit(T record) throws IOException, InterruptedException
RecordWriter
randomEmit
in class RecordWriter<T extends IOReadableWritable>
IOException
InterruptedException
public void broadcastEmit(T record) throws IOException, InterruptedException
RecordWriter
broadcastEmit
in class RecordWriter<T extends IOReadableWritable>
IOException
InterruptedException
public void flushTargetPartition(int targetChannel)
randomEmit(IOReadableWritable)
, emit(IOReadableWritable)
or broadcastEmit(IOReadableWritable)
. Only random emit
should flush a single target channel, otherwise we should flush all the channels.flushTargetPartition
in class RecordWriter<T extends IOReadableWritable>
public BufferBuilder getBufferBuilder(int targetChannel) throws IOException, InterruptedException
RecordWriter
BufferBuilder
may already exist if not filled up last time, otherwise we need
request a new one for this target channel.IOException
InterruptedException
public BufferBuilder requestNewBufferBuilder(int targetChannel) throws IOException, InterruptedException
randomEmit(IOReadableWritable)
.
For non-broadcast, the created BufferConsumer
is only for the target channel.
For broadcast, all the channels share the same requested BufferBuilder
and the
created BufferConsumer
is copied for every channel.
requestNewBufferBuilder
in class RecordWriter<T extends IOReadableWritable>
IOException
InterruptedException
public void tryFinishCurrentBufferBuilder(int targetChannel)
RecordWriter
BufferBuilder
as finished if present and clears the state for next
one.public void emptyCurrentBufferBuilder(int targetChannel)
RecordWriter
BufferBuilder
as empty for the target channel.public void closeBufferBuilder(int targetChannel)
RecordWriter
BufferBuilder
as finished and releases the resources for the target
channel.public void clearBuffers()
RecordWriter
BufferBuilder
s for all the channels.clearBuffers
in class RecordWriter<T extends IOReadableWritable>
Copyright © 2014–2021 The Apache Software Foundation. All rights reserved.