@Internal public class ContinuousFileReaderOperator<OUT,S extends Serializable> extends AbstractStreamOperator<OUT> implements OneInputStreamOperator<FileInputSplit,OUT>, OutputTypeConfigurable<OUT>, AsyncExceptionChecker
FileInputSplits
received from
the preceding ContinuousFileMonitoringFunction
. This operator can have parallelism
greater than 1, contrary to the ContinuousFileMonitoringFunction
which has
a parallelism of 1.
This operator will receive the split descriptors, put them in a queue, and have another
thread read the actual data from the split. This architecture allows the separation of the
reading thread, from the one emitting the checkpoint barriers, thus removing any potential
back-pressure.AbstractStreamOperator.CountingOutput
chainingStrategy, metrics, output
Constructor and Description |
---|
ContinuousFileReaderOperator(FileInputFormat<OUT> format) |
Modifier and Type | Method and Description |
---|---|
void |
checkAsyncException()
Checks if an asynchronous exception was thrown.
|
void |
close()
This method is called after all records have been added to the operators via the methods
OneInputStreamOperator.processElement(StreamRecord) , or
TwoInputStreamOperator.processElement1(StreamRecord) and
TwoInputStreamOperator.processElement2(StreamRecord) . |
void |
dispose()
This method is called at the very end of the operator's life, both in the case of a successful
completion of the operation, and in the case of a failure and canceling.
|
void |
open()
This method is called immediately before any elements are processed, it should contain the
operator's initialization logic.
|
void |
processElement(StreamRecord<FileInputSplit> element)
Processes one element that arrived at this operator.
|
void |
processWatermark(Watermark mark)
Processes a
Watermark . |
void |
restoreState(StreamTaskState state)
Restores the operator state, if this operator's execution is recovering from a checkpoint.
|
void |
setOutputType(TypeInformation<OUT> outTypeInfo,
ExecutionConfig executionConfig)
Is called by the
StreamGraph.addOperator(Integer, String, StreamOperator, TypeInformation, TypeInformation, String)
method when the StreamGraph is generated. |
StreamTaskState |
snapshotOperatorState(long checkpointId,
long timestamp)
Called to draw a state snapshot from the operator.
|
getChainingStrategy, getContainingTask, getCurrentProcessingTime, getExecutionConfig, getMetricGroup, getOperatorConfig, getOperatorName, getPartitionedState, getPartitionedState, getRuntimeContext, getStateBackend, getUserCodeClassloader, notifyOfCompletedCheckpoint, registerTimer, setChainingStrategy, setKeyContext, setKeyContextElement1, setKeyContextElement2, setup
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
getChainingStrategy, getMetricGroup, notifyOfCompletedCheckpoint, setChainingStrategy, setKeyContextElement1, setKeyContextElement2, setup
public ContinuousFileReaderOperator(FileInputFormat<OUT> format)
public void setOutputType(TypeInformation<OUT> outTypeInfo, ExecutionConfig executionConfig)
OutputTypeConfigurable
StreamGraph.addOperator(Integer, String, StreamOperator, TypeInformation, TypeInformation, String)
method when the StreamGraph
is generated. The
method is called with the output TypeInformation
which is also used for the
StreamTask
output serializer.setOutputType
in interface OutputTypeConfigurable<OUT>
outTypeInfo
- Output type information of the StreamTask
executionConfig
- Execution configurationpublic void open() throws Exception
AbstractStreamOperator
The default implementation does nothing.
open
in interface StreamOperator<OUT>
open
in class AbstractStreamOperator<OUT>
Exception
- An exception in this method causes the operator to fail.public void processElement(StreamRecord<FileInputSplit> element) throws Exception
OneInputStreamOperator
processElement
in interface OneInputStreamOperator<FileInputSplit,OUT>
Exception
public void processWatermark(Watermark mark) throws Exception
OneInputStreamOperator
Watermark
.
This method is guaranteed to not be called concurrently with other methods of the operator.processWatermark
in interface OneInputStreamOperator<FileInputSplit,OUT>
Exception
Watermark
public void dispose()
AbstractStreamOperator
dispose
in interface StreamOperator<OUT>
dispose
in class AbstractStreamOperator<OUT>
public void close() throws Exception
AbstractStreamOperator
OneInputStreamOperator.processElement(StreamRecord)
, or
TwoInputStreamOperator.processElement1(StreamRecord)
and
TwoInputStreamOperator.processElement2(StreamRecord)
.
The method is expected to flush all remaining buffered data. Exceptions during this flushing of buffered should be propagated, in order to cause the operation to be recognized asa failed, because the last data items are not processed properly.
close
in interface StreamOperator<OUT>
close
in class AbstractStreamOperator<OUT>
Exception
- An exception in this method causes the operator to fail.public void checkAsyncException()
AsyncExceptionChecker
checkAsyncException
in interface AsyncExceptionChecker
public StreamTaskState snapshotOperatorState(long checkpointId, long timestamp) throws Exception
StreamOperator
snapshotOperatorState
in interface StreamOperator<OUT>
snapshotOperatorState
in class AbstractStreamOperator<OUT>
checkpointId
- The ID of the checkpoint.timestamp
- The timestamp of the checkpoint.Exception
- Forwards exceptions that occur while drawing snapshots from the operator
and the key/value state.public void restoreState(StreamTaskState state) throws Exception
StreamOperator
This method is called after StreamOperator.setup(StreamTask, StreamConfig, Output)
and before StreamOperator.open()
.
restoreState
in interface StreamOperator<OUT>
restoreState
in class AbstractStreamOperator<OUT>
state
- The state of operator that was snapshotted as part of checkpoint
from which the execution is restored.Exception
- Exceptions during state restore should be forwarded, so that the system can
properly react to failed state restore and fail the execution attempt.Copyright © 2014–2017 The Apache Software Foundation. All rights reserved.