public abstract class FlinkKafkaConsumerBase<T> extends RichParallelSourceFunction<T> implements CheckpointListener, CheckpointedAsynchronously<HashMap<KafkaTopicPartition,Long>>, ResultTypeQueryable<T>
SourceFunction.SourceContext<T>
Modifier and Type | Field and Description |
---|---|
protected KeyedDeserializationSchema<T> |
deserializer
The schema to convert between Kafka#s byte messages, and Flink's objects
|
static int |
MAX_NUM_PENDING_CHECKPOINTS
The maximum number of pending non-committed checkpoints to track, to avoid memory leaks
|
protected HashMap<KafkaTopicPartition,Long> |
offsetsState
The offsets of the last returned elements
|
protected org.apache.commons.collections.map.LinkedMap |
pendingCheckpoints
Data for pending but uncommitted checkpoints
|
protected HashMap<KafkaTopicPartition,Long> |
restoreToOffset
The offsets to restore to, if the consumer restores state from a checkpoint
|
protected boolean |
running
Flag indicating whether the consumer is still running
|
Constructor and Description |
---|
FlinkKafkaConsumerBase(KeyedDeserializationSchema<T> deserializer,
Properties props)
Creates a new Flink Kafka Consumer, using the given type of fetcher and offset handler.
|
Modifier and Type | Method and Description |
---|---|
protected static <T> List<T> |
assignPartitions(List<T> partitions,
int numConsumers,
int consumerIndex) |
protected abstract void |
commitOffsets(HashMap<KafkaTopicPartition,Long> checkpointOffsets) |
TypeInformation<T> |
getProducedType()
Gets the data type (as a
TypeInformation ) produced by this function or input format. |
static void |
logPartitionInfo(List<KafkaTopicPartition> partitionInfos)
Method to log partition information.
|
void |
notifyCheckpointComplete(long checkpointId)
This method is called as a notification once a distributed checkpoint has been completed.
|
void |
restoreState(HashMap<KafkaTopicPartition,Long> restoredOffsets)
Restores the state of the function or operator to that of a previous checkpoint.
|
HashMap<KafkaTopicPartition,Long> |
snapshotState(long checkpointId,
long checkpointTimestamp)
Gets the current state of the function of operator.
|
close, getIterationRuntimeContext, getRuntimeContext, open, setRuntimeContext
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
cancel, run
public static final int MAX_NUM_PENDING_CHECKPOINTS
protected final KeyedDeserializationSchema<T> deserializer
protected final org.apache.commons.collections.map.LinkedMap pendingCheckpoints
protected transient HashMap<KafkaTopicPartition,Long> offsetsState
protected transient HashMap<KafkaTopicPartition,Long> restoreToOffset
protected volatile boolean running
public FlinkKafkaConsumerBase(KeyedDeserializationSchema<T> deserializer, Properties props)
To determine which kink of fetcher and offset handler to use, please refer to the docs at the beginnign of this class.
deserializer
- The deserializer to turn raw byte messages into Java/Scala objects.props
- The properties that are used to configure both the fetcher and the offset handler.public HashMap<KafkaTopicPartition,Long> snapshotState(long checkpointId, long checkpointTimestamp) throws Exception
Checkpointed
snapshotState
in interface Checkpointed<HashMap<KafkaTopicPartition,Long>>
checkpointId
- The ID of the checkpoint.checkpointTimestamp
- The timestamp of the checkpoint, as derived by
System.currentTimeMillis() on the JobManager.Exception
- Thrown if the creation of the state object failed. This causes the
checkpoint to fail. The system may decide to fail the operation (and trigger
recovery), or to discard this checkpoint attempt and to continue running
and to try again with the next checkpoint attempt.public void restoreState(HashMap<KafkaTopicPartition,Long> restoredOffsets)
Checkpointed
restoreState
in interface Checkpointed<HashMap<KafkaTopicPartition,Long>>
restoredOffsets
- The state to be restored.public void notifyCheckpointComplete(long checkpointId) throws Exception
CheckpointListener
notifyCheckpointComplete
in interface CheckpointListener
checkpointId
- The ID of the checkpoint that has been completed.Exception
protected abstract void commitOffsets(HashMap<KafkaTopicPartition,Long> checkpointOffsets) throws Exception
Exception
public TypeInformation<T> getProducedType()
ResultTypeQueryable
TypeInformation
) produced by this function or input format.getProducedType
in interface ResultTypeQueryable<T>
protected static <T> List<T> assignPartitions(List<T> partitions, int numConsumers, int consumerIndex)
public static void logPartitionInfo(List<KafkaTopicPartition> partitionInfos)
partitionInfos
- List of subscribed partitionsCopyright © 2014–2017 The Apache Software Foundation. All rights reserved.