org.apache.flink.state.api.output.operators

Class StateBootstrapWrapperOperator<IN,OUT,OP extends AbstractStreamOperator<OUT> & OneInputStreamOperator<IN,OUT>>

java.lang.Object

org.apache.flink.state.api.output.operators.StateBootstrapWrapperOperator<IN,OUT,OP>

All Implemented Interfaces:

Serializable, CheckpointListener, BoundedOneInput, Input<IN>, KeyContext, OneInputStreamOperator<IN,TaggedOperatorSubtaskState>, SetupableStreamOperator<TaggedOperatorSubtaskState>, StreamOperator<TaggedOperatorSubtaskState>, Disposable

@Internal
public final class StateBootstrapWrapperOperator<IN,OUT,OP extends AbstractStreamOperator<OUT> & OneInputStreamOperator<IN,OUT>>
extends Object
implements OneInputStreamOperator<IN,TaggedOperatorSubtaskState>, SetupableStreamOperator<TaggedOperatorSubtaskState>, BoundedOneInput

Wraps an existing operator so it can be bootstrapped.

See Also:

Serialized Form

Constructor Summary

Constructors

Constructor and Description
StateBootstrapWrapperOperator(long timestamp, Path savepointPath, OP operator)

Method Summary

Modifier and Type	Method and Description
void	close() This method is called after all records have been added to the operators via the methods Input.processElement(StreamRecord), or TwoInputStreamOperator.processElement1(StreamRecord) and TwoInputStreamOperator.processElement2(StreamRecord).
void	dispose() This method is called at the very end of the operator's life, both in the case of a successful completion of the operation, and in the case of a failure and canceling.
void	endInput() It is notified that no more data will arrive from the input.
ChainingStrategy	getChainingStrategy()
Object	getCurrentKey()
MetricGroup	getMetricGroup()
OperatorID	getOperatorID()
void	initializeState(StreamTaskStateInitializer streamTaskStateManager) Provides a context to initialize all state in the operator.
void	notifyCheckpointComplete(long checkpointId) Notifies the listener that the checkpoint with the given checkpointId completed and was committed.
void	open() This method is called immediately before any elements are processed, it should contain the operator's initialization logic.
void	prepareSnapshotPreBarrier(long checkpointId) This method is called when the operator should do a snapshot, before it emits its own checkpoint barrier.
void	processElement(StreamRecord<IN> element) Processes one element that arrived on this input of the MultipleInputStreamOperator.
void	processLatencyMarker(LatencyMarker latencyMarker) Processes a LatencyMarker that arrived on the first input of this two-input operator.
void	processWatermark(Watermark mark) Processes a Watermark that arrived on the first input of this two-input operator.
void	setChainingStrategy(ChainingStrategy strategy)
void	setCurrentKey(Object key)
void	setKeyContextElement1(StreamRecord<?> record)
void	setKeyContextElement2(StreamRecord<?> record)
void	setup(StreamTask<?,?> containingTask, StreamConfig config, Output<StreamRecord<TaggedOperatorSubtaskState>> output) Initializes the operator.
OperatorSnapshotFutures	snapshotState(long checkpointId, long timestamp, CheckpointOptions checkpointOptions, CheckpointStreamFactory storageLocation) Called to draw a state snapshot from the operator.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.flink.streaming.api.operators.OneInputStreamOperator

setKeyContextElement

Methods inherited from interface org.apache.flink.api.common.state.CheckpointListener

notifyCheckpointAborted

Constructor Detail

StateBootstrapWrapperOperator

public StateBootstrapWrapperOperator(long timestamp,
                                     Path savepointPath,
                                     OP operator)

Method Detail

setup

public void setup(StreamTask<?,?> containingTask,
                  StreamConfig config,
                  Output<StreamRecord<TaggedOperatorSubtaskState>> output)

Description copied from interface: SetupableStreamOperator

Initializes the operator. Sets access to the context and the output.

Specified by:

setup in interface SetupableStreamOperator<TaggedOperatorSubtaskState>

processElement

public void processElement(StreamRecord<IN> element)
                    throws Exception

Description copied from interface: Input

Processes one element that arrived on this input of the MultipleInputStreamOperator. This method is guaranteed to not be called concurrently with other methods of the operator.

Specified by:

processElement in interface Input<IN>

Throws:

Exception

processWatermark

public void processWatermark(Watermark mark)
                      throws Exception

Description copied from interface: Input

Processes a Watermark that arrived on the first input of this two-input operator. This method is guaranteed to not be called concurrently with other methods of the operator.

Specified by:

processWatermark in interface Input<IN>

Throws:

Exception

See Also:

Watermark

processLatencyMarker

public void processLatencyMarker(LatencyMarker latencyMarker)
                          throws Exception

Description copied from interface: Input

Processes a LatencyMarker that arrived on the first input of this two-input operator. This method is guaranteed to not be called concurrently with other methods of the operator.

Specified by:

processLatencyMarker in interface Input<IN>

Throws:

Exception

See Also:

LatencyMarker

open

public void open()
          throws Exception

Description copied from interface: StreamOperator

This method is called immediately before any elements are processed, it should contain the operator's initialization logic.

Specified by:

open in interface StreamOperator<TaggedOperatorSubtaskState>

Throws:

Exception - An exception in this method causes the operator to fail.

close

public void close()
           throws Exception

Description copied from interface: StreamOperator

This method is called after all records have been added to the operators via the methods Input.processElement(StreamRecord), or TwoInputStreamOperator.processElement1(StreamRecord) and TwoInputStreamOperator.processElement2(StreamRecord).

The method is expected to flush all remaining buffered data. Exceptions during this flushing of buffered should be propagated, in order to cause the operation to be recognized as failed, because the last data items are not processed properly.

Specified by:

close in interface StreamOperator<TaggedOperatorSubtaskState>

Throws:

Exception - An exception in this method causes the operator to fail.

dispose

public void dispose()
             throws Exception

Description copied from interface: StreamOperator

This method is called at the very end of the operator's life, both in the case of a successful completion of the operation, and in the case of a failure and canceling.

This method is expected to make a thorough effort to release all resources that the operator has acquired.

Specified by:

dispose in interface StreamOperator<TaggedOperatorSubtaskState>

Specified by:

dispose in interface Disposable

Throws:

Exception - if something goes wrong during disposal.

prepareSnapshotPreBarrier

public void prepareSnapshotPreBarrier(long checkpointId)
                               throws Exception

Description copied from interface: StreamOperator

This method is called when the operator should do a snapshot, before it emits its own checkpoint barrier.

This method is intended not for any actual state persistence, but only for emitting some data before emitting the checkpoint barrier. Operators that maintain some small transient state that is inefficient to checkpoint (especially when it would need to be checkpointed in a re-scalable way) but can simply be sent downstream before the checkpoint. An example are opportunistic pre-aggregation operators, which have small the pre-aggregation state that is frequently flushed downstream.

Important: This method should not be used for any actual state snapshot logic, because it will inherently be within the synchronous part of the operator's checkpoint. If heavy work is done within this method, it will affect latency and downstream checkpoint alignments.

Specified by:

prepareSnapshotPreBarrier in interface StreamOperator<TaggedOperatorSubtaskState>

Parameters:

checkpointId - The ID of the checkpoint.

Throws:

Exception - Throwing an exception here causes the operator to fail and go into recovery.

snapshotState

public OperatorSnapshotFutures snapshotState(long checkpointId,
                                             long timestamp,
                                             CheckpointOptions checkpointOptions,
                                             CheckpointStreamFactory storageLocation)
                                      throws Exception

Description copied from interface: StreamOperator

Called to draw a state snapshot from the operator.

Specified by:

snapshotState in interface StreamOperator<TaggedOperatorSubtaskState>

Returns:

a runnable future to the state handle that points to the snapshotted state. For synchronous implementations, the runnable might already be finished.

Throws:

Exception - exception that happened during snapshotting.

initializeState

public void initializeState(StreamTaskStateInitializer streamTaskStateManager)
                     throws Exception

Description copied from interface: StreamOperator

Provides a context to initialize all state in the operator.

Specified by:

initializeState in interface StreamOperator<TaggedOperatorSubtaskState>

Throws:

Exception

setKeyContextElement1

public void setKeyContextElement1(StreamRecord<?> record)
                           throws Exception

Specified by:

setKeyContextElement1 in interface StreamOperator<TaggedOperatorSubtaskState>

Throws:

Exception

setKeyContextElement2

public void setKeyContextElement2(StreamRecord<?> record)
                           throws Exception

Specified by:

setKeyContextElement2 in interface StreamOperator<TaggedOperatorSubtaskState>

Throws:

Exception

getChainingStrategy

public ChainingStrategy getChainingStrategy()

Specified by:

getChainingStrategy in interface SetupableStreamOperator<TaggedOperatorSubtaskState>

setChainingStrategy

public void setChainingStrategy(ChainingStrategy strategy)

Specified by:

setChainingStrategy in interface SetupableStreamOperator<TaggedOperatorSubtaskState>

getMetricGroup

public MetricGroup getMetricGroup()

Specified by:

getMetricGroup in interface StreamOperator<TaggedOperatorSubtaskState>

getOperatorID

public OperatorID getOperatorID()

Specified by:

getOperatorID in interface StreamOperator<TaggedOperatorSubtaskState>

notifyCheckpointComplete

public void notifyCheckpointComplete(long checkpointId)
                              throws Exception

Description copied from interface: CheckpointListener

Notifies the listener that the checkpoint with the given checkpointId completed and was committed.

These notifications are "best effort", meaning they can sometimes be skipped. To behave properly, implementers need to follow the "Checkpoint Subsuming Contract". Please see the class-level JavaDocs for details.

Please note that checkpoints may generally overlap, so you cannot assume that the notifyCheckpointComplete() call is always for the latest prior checkpoint (or snapshot) that was taken on the function/operator implementing this interface. It might be for a checkpoint that was triggered earlier. Implementing the "Checkpoint Subsuming Contract" (see above) properly handles this situation correctly as well.

Please note that throwing exceptions from this method will not cause the completed checkpoint to be revoked. Throwing exceptions will typically cause task/job failure and trigger recovery.

Specified by:

notifyCheckpointComplete in interface CheckpointListener

Parameters:

checkpointId - The ID of the checkpoint that has been completed.

Throws:

Exception - This method can propagate exceptions, which leads to a failure/recovery for the task. Not that this will NOT lead to the checkpoint being revoked.

setCurrentKey

public void setCurrentKey(Object key)

Specified by:

setCurrentKey in interface KeyContext

getCurrentKey

public Object getCurrentKey()

Specified by:

getCurrentKey in interface KeyContext

endInput

public void endInput()
              throws Exception

Description copied from interface: BoundedOneInput

It is notified that no more data will arrive from the input.

WARNING: It is not safe to use this method to commit any transactions or other side effects! You can use this method to flush any buffered data that can later on be committed e.g. in a CheckpointListener.notifyCheckpointComplete(long).

Specified by:

endInput in interface BoundedOneInput

Throws:

Exception