@Experimental public class DoubleEmittingSourceReaderWithCheckpointsInBetween<E,O,IterT extends Iterator<E>,SplitT extends IteratorSourceSplit<E,IterT>> extends IteratorSourceReaderBase<E,O,IterT,SplitT>
SourceReader
that synchronizes emission of N elements on the arrival of the checkpoint
barriers It 1) emits a list of elements without checkpoints in-between, 2) then waits for two
checkpoints to complete, 3) then re-emits the same elements before 4) waiting for another two
checkpoints and 5) exiting.
This lockstep execution is possible because pollNext
and snapshotState
are
executed in the same thread and the fact that pollNext
can emit N elements at once. This
reader is meant to be used solely for testing purposes as the substitution for the FiniteTestSource
which implements the deprecated SourceFunction
API.
availability, currentSplit, iterator
Constructor and Description |
---|
DoubleEmittingSourceReaderWithCheckpointsInBetween(SourceReaderContext context,
GeneratorFunction<E,O> generatorFunction) |
DoubleEmittingSourceReaderWithCheckpointsInBetween(SourceReaderContext context,
GeneratorFunction<E,O> generatorFunction,
java.util.function.BooleanSupplier allowedToExit) |
Modifier and Type | Method and Description |
---|---|
protected O |
convert(E value) |
void |
notifyCheckpointComplete(long checkpointId)
We have an empty default implementation here because most source readers do not have to
implement the method.
|
InputStatus |
pollNext(ReaderOutput<O> output)
Poll the next available record into the
ReaderOutput . |
void |
start(SourceReaderContext context) |
addSplits, close, finishSplit, isAvailable, notifyNoMoreSplits, snapshotState, start, tryMoveToNextSplit
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
handleSourceEvents, pauseOrResumeSplits
notifyCheckpointAborted
public DoubleEmittingSourceReaderWithCheckpointsInBetween(SourceReaderContext context, GeneratorFunction<E,O> generatorFunction, @Nullable java.util.function.BooleanSupplier allowedToExit)
public DoubleEmittingSourceReaderWithCheckpointsInBetween(SourceReaderContext context, GeneratorFunction<E,O> generatorFunction)
public void start(SourceReaderContext context)
public InputStatus pollNext(ReaderOutput<O> output)
SourceReader
ReaderOutput
.
The implementation must make sure this method is non-blocking.
Although the implementation can emit multiple records into the given ReaderOutput, it is
recommended not doing so. Instead, emit one record into the ReaderOutput and return a InputStatus.MORE_AVAILABLE
to let the caller thread know there are more records available.
pollNext
in interface SourceReader<O,SplitT extends IteratorSourceSplit<E,IterT>>
pollNext
in class IteratorSourceReaderBase<E,O,IterT extends Iterator<E>,SplitT extends IteratorSourceSplit<E,IterT>>
public void notifyCheckpointComplete(long checkpointId) throws Exception
SourceReader
checkpointId
- The ID of the checkpoint that has been completed.Exception
- This method can propagate exceptions, which leads to a failure/recovery for
the task. Note that this will NOT lead to the checkpoint being revoked.CheckpointListener.notifyCheckpointComplete(long)
Copyright © 2014–2024 The Apache Software Foundation. All rights reserved.