T
- The type of elements produced by the fetcher.public class Kafka08Fetcher<T> extends AbstractFetcher<T,kafka.common.TopicAndPartition>
checkpointLock, NO_TIMESTAMPS_WATERMARKS, PERIODIC_WATERMARKS, PUNCTUATED_WATERMARKS, sourceContext, timestampWatermarkMode, useMetrics
Constructor and Description |
---|
Kafka08Fetcher(SourceFunction.SourceContext<T> sourceContext,
Map<KafkaTopicPartition,Long> assignedPartitionsWithInitialOffsets,
SerializedValue<AssignerWithPeriodicWatermarks<T>> watermarksPeriodic,
SerializedValue<AssignerWithPunctuatedWatermarks<T>> watermarksPunctuated,
StreamingRuntimeContext runtimeContext,
KeyedDeserializationSchema<T> deserializer,
Properties kafkaProperties,
long autoCommitInterval,
boolean useMetrics) |
Modifier and Type | Method and Description |
---|---|
void |
cancel() |
void |
commitInternalOffsetsToKafka(Map<KafkaTopicPartition,Long> offsets,
KafkaCommitCallback commitCallback)
Commits the given partition offsets to the Kafka brokers (or to ZooKeeper for
older Kafka versions).
|
kafka.common.TopicAndPartition |
createKafkaPartitionHandle(KafkaTopicPartition partition)
Creates the Kafka version specific representation of the given
topic partition.
|
void |
runFetchLoop() |
addOffsetStateGauge, emitRecord, emitRecordWithTimestamp, emitRecordWithTimestampAndPeriodicWatermark, emitRecordWithTimestampAndPunctuatedWatermark, snapshotCurrentState, subscribedPartitionStates
public Kafka08Fetcher(SourceFunction.SourceContext<T> sourceContext, Map<KafkaTopicPartition,Long> assignedPartitionsWithInitialOffsets, SerializedValue<AssignerWithPeriodicWatermarks<T>> watermarksPeriodic, SerializedValue<AssignerWithPunctuatedWatermarks<T>> watermarksPunctuated, StreamingRuntimeContext runtimeContext, KeyedDeserializationSchema<T> deserializer, Properties kafkaProperties, long autoCommitInterval, boolean useMetrics) throws Exception
Exception
public void runFetchLoop() throws Exception
runFetchLoop
in class AbstractFetcher<T,kafka.common.TopicAndPartition>
Exception
public void cancel()
cancel
in class AbstractFetcher<T,kafka.common.TopicAndPartition>
public kafka.common.TopicAndPartition createKafkaPartitionHandle(KafkaTopicPartition partition)
AbstractFetcher
createKafkaPartitionHandle
in class AbstractFetcher<T,kafka.common.TopicAndPartition>
partition
- The Flink representation of the Kafka topic partition.public void commitInternalOffsetsToKafka(Map<KafkaTopicPartition,Long> offsets, @Nonnull KafkaCommitCallback commitCallback) throws Exception
AbstractFetcher
OffsetCommitMode.ON_CHECKPOINTS
.
The given offsets are the internal checkpointed offsets, representing
the last processed record of each partition. Version-specific implementations of this method
need to hold the contract that the given offsets must be incremented by 1 before
committing them, so that committed offsets to Kafka represent "the next record to process".commitInternalOffsetsToKafka
in class AbstractFetcher<T,kafka.common.TopicAndPartition>
offsets
- The offsets to commit to Kafka (implementations must increment offsets by 1 before committing).commitCallback
- The callback that the user should trigger when a commit request completes or fails.Exception
- This method forwards exceptions.Copyright © 2014–2018 The Apache Software Foundation. All rights reserved.