public class StreamingFileWriter extends AbstractStreamOperator<StreamingFileCommitter.CommitMessage> implements OneInputStreamOperator<RowData,StreamingFileCommitter.CommitMessage>, BoundedOneInput
StreamingFileSink
. It sends
partition commit message to downstream for committing.
chainingStrategy, config, latencyStats, LOG, metrics, output, processingTimeService
Constructor and Description |
---|
StreamingFileWriter(long bucketCheckInterval,
StreamingFileSink.BucketsBuilder<RowData,String,? extends StreamingFileSink.BucketsBuilder<RowData,String,?>> bucketsBuilder) |
Modifier and Type | Method and Description |
---|---|
void |
dispose()
This method is called at the very end of the operator's life, both in the case of a
successful completion of the operation, and in the case of a failure and canceling.
|
void |
endInput()
It is notified that no more data will arrive on the input.
|
void |
initializeState(StateInitializationContext context)
Stream operators with state which can be restored need to override this hook method.
|
void |
notifyCheckpointComplete(long checkpointId)
Commit up to this checkpoint id, also send inactive partitions to downstream for committing.
|
void |
processElement(StreamRecord<RowData> element)
Processes one element that arrived at this operator.
|
void |
processWatermark(Watermark mark)
Processes a
Watermark . |
void |
snapshotState(StateSnapshotContext context)
Stream operators with state, which want to participate in a snapshot need to override this
hook method.
|
close, getChainingStrategy, getContainingTask, getCurrentKey, getExecutionConfig, getInternalTimerService, getKeyedStateBackend, getKeyedStateStore, getMetricGroup, getOperatorConfig, getOperatorID, getOperatorName, getOperatorStateBackend, getOrCreateKeyedState, getPartitionedState, getPartitionedState, getProcessingTimeService, getRuntimeContext, getTimeServiceManager, getUserCodeClassloader, initializeState, isUsingCustomRawKeyedState, notifyCheckpointAborted, numEventTimeTimers, numProcessingTimeTimers, open, prepareSnapshotPreBarrier, processLatencyMarker, processLatencyMarker1, processLatencyMarker2, processWatermark1, processWatermark2, reportOrForwardLatencyMarker, setChainingStrategy, setCurrentKey, setKeyContextElement1, setKeyContextElement2, setProcessingTimeService, setup, snapshotState
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
processLatencyMarker
close, getMetricGroup, getOperatorID, initializeState, open, prepareSnapshotPreBarrier, setKeyContextElement1, setKeyContextElement2, snapshotState
notifyCheckpointAborted
getCurrentKey, setCurrentKey
public StreamingFileWriter(long bucketCheckInterval, StreamingFileSink.BucketsBuilder<RowData,String,? extends StreamingFileSink.BucketsBuilder<RowData,String,?>> bucketsBuilder)
public void initializeState(StateInitializationContext context) throws Exception
AbstractStreamOperator
initializeState
in interface StreamOperatorStateHandler.CheckpointedStreamOperator
initializeState
in class AbstractStreamOperator<StreamingFileCommitter.CommitMessage>
context
- context that allows to register different states.Exception
public void snapshotState(StateSnapshotContext context) throws Exception
AbstractStreamOperator
snapshotState
in interface StreamOperatorStateHandler.CheckpointedStreamOperator
snapshotState
in class AbstractStreamOperator<StreamingFileCommitter.CommitMessage>
context
- context that provides information and means required for taking a snapshotException
public void processWatermark(Watermark mark) throws Exception
OneInputStreamOperator
Watermark
. This method is guaranteed to not be called concurrently with
other methods of the operator.processWatermark
in interface OneInputStreamOperator<RowData,StreamingFileCommitter.CommitMessage>
processWatermark
in class AbstractStreamOperator<StreamingFileCommitter.CommitMessage>
Exception
Watermark
public void processElement(StreamRecord<RowData> element) throws Exception
OneInputStreamOperator
processElement
in interface OneInputStreamOperator<RowData,StreamingFileCommitter.CommitMessage>
Exception
public void notifyCheckpointComplete(long checkpointId) throws Exception
notifyCheckpointComplete
in interface CheckpointListener
notifyCheckpointComplete
in class AbstractStreamOperator<StreamingFileCommitter.CommitMessage>
checkpointId
- The ID of the checkpoint that has been completed.Exception
- This method can propagate exceptions, which leads to a failure/recovery for
the task. Not that this will NOT lead to the checkpoint being revoked.public void endInput() throws Exception
BoundedOneInput
endInput
in interface BoundedOneInput
Exception
public void dispose() throws Exception
AbstractStreamOperator
This method is expected to make a thorough effort to release all resources that the operator has acquired.
dispose
in interface StreamOperator<StreamingFileCommitter.CommitMessage>
dispose
in interface Disposable
dispose
in class AbstractStreamOperator<StreamingFileCommitter.CommitMessage>
Exception
- if something goes wrong during disposal.Copyright © 2014–2021 The Apache Software Foundation. All rights reserved.