T
- The type of elements deserialized from Kafka's byte records, and emitted into
the Flink data streams.KPH
- The type of topic/partition identifier used by Kafka in the specific version.public abstract class AbstractFetcher<T,KPH> extends Object
This fetcher base class implements the logic around emitting records and tracking offsets, as well as around the optional timestamp assignment and watermark generation.
Modifier and Type | Field and Description |
---|---|
protected boolean |
useMetrics
Flag whether to register metrics for the fetcher
|
Modifier | Constructor and Description |
---|---|
protected |
AbstractFetcher(SourceFunction.SourceContext<T> sourceContext,
List<KafkaTopicPartition> assignedPartitions,
SerializedValue<AssignerWithPeriodicWatermarks<T>> watermarksPeriodic,
SerializedValue<AssignerWithPunctuatedWatermarks<T>> watermarksPunctuated,
StreamingRuntimeContext runtimeContext,
boolean useMetrics) |
Modifier and Type | Method and Description |
---|---|
protected void |
addOffsetStateGauge(MetricGroup metricGroup)
Add current and committed offsets to metric group
|
abstract void |
cancel() |
abstract void |
commitSpecificOffsetsToKafka(Map<KafkaTopicPartition,Long> offsets)
Commits the given partition offsets to the Kafka brokers (or to ZooKeeper for
older Kafka versions).
|
abstract KPH |
createKafkaPartitionHandle(KafkaTopicPartition partition)
Creates the Kafka version specific representation of the given
topic partition.
|
protected void |
emitRecord(T record,
KafkaTopicPartitionState<KPH> partitionState,
long offset)
Implementation Note: This method is kept brief to be JIT inlining friendly.
|
void |
restoreOffsets(Map<KafkaTopicPartition,Long> restoredOffsets)
Restores the partition offsets.
|
abstract void |
runFetchLoop() |
HashMap<KafkaTopicPartition,Long> |
snapshotCurrentState()
Takes a snapshot of the partition offsets.
|
protected KafkaTopicPartitionState<KPH>[] |
subscribedPartitions()
Gets all partitions (with partition state) that this fetcher is subscribed to.
|
protected final boolean useMetrics
protected AbstractFetcher(SourceFunction.SourceContext<T> sourceContext, List<KafkaTopicPartition> assignedPartitions, SerializedValue<AssignerWithPeriodicWatermarks<T>> watermarksPeriodic, SerializedValue<AssignerWithPunctuatedWatermarks<T>> watermarksPunctuated, StreamingRuntimeContext runtimeContext, boolean useMetrics) throws Exception
Exception
protected final KafkaTopicPartitionState<KPH>[] subscribedPartitions()
public abstract void cancel()
public abstract KPH createKafkaPartitionHandle(KafkaTopicPartition partition)
partition
- The Flink representation of the Kafka topic partition.public abstract void commitSpecificOffsetsToKafka(Map<KafkaTopicPartition,Long> offsets) throws Exception
offsets
- The offsets to commit to Kafka.Exception
- This method forwards exceptions.public HashMap<KafkaTopicPartition,Long> snapshotCurrentState()
Important: This method mus be called under the checkpoint lock.
public void restoreOffsets(Map<KafkaTopicPartition,Long> restoredOffsets)
restoredOffsets
- The restored offsets for the partitionsIllegalStateException
- if the partitions in the provided restored offsets map
cannot completely match the fetcher's subscribed partitions.protected final void emitRecord(T record, KafkaTopicPartitionState<KPH> partitionState, long offset)
Implementation Note: This method is kept brief to be JIT inlining friendly. That makes the fast path efficient, the extended paths are called as separate methods.
record
- The record to emitpartitionState
- The state of the Kafka partition from which the record was fetchedoffset
- The offset from which the record was fetchedprotected void addOffsetStateGauge(MetricGroup metricGroup)
metricGroup
- The metric group to useCopyright © 2014–2017 The Apache Software Foundation. All rights reserved.