T
- The type of elements produced by the fetcher.public class Kafka09Fetcher<T> extends AbstractFetcher<T,org.apache.kafka.common.TopicPartition>
checkpointLock, NO_TIMESTAMPS_WATERMARKS, PERIODIC_WATERMARKS, PUNCTUATED_WATERMARKS, sourceContext, timestampWatermarkMode, useMetrics
Constructor and Description |
---|
Kafka09Fetcher(SourceFunction.SourceContext<T> sourceContext,
List<KafkaTopicPartition> assignedPartitions,
SerializedValue<AssignerWithPeriodicWatermarks<T>> watermarksPeriodic,
SerializedValue<AssignerWithPunctuatedWatermarks<T>> watermarksPunctuated,
ProcessingTimeService processingTimeProvider,
long autoWatermarkInterval,
ClassLoader userCodeClassLoader,
boolean enableCheckpointing,
String taskNameWithSubtasks,
MetricGroup metricGroup,
KeyedDeserializationSchema<T> deserializer,
Properties kafkaProperties,
long pollTimeout,
boolean useMetrics) |
Modifier and Type | Method and Description |
---|---|
void |
cancel() |
void |
commitInternalOffsetsToKafka(Map<KafkaTopicPartition,Long> offsets)
Commits the given partition offsets to the Kafka brokers (or to ZooKeeper for
older Kafka versions).
|
protected KafkaConsumerCallBridge |
createCallBridge() |
org.apache.kafka.common.TopicPartition |
createKafkaPartitionHandle(KafkaTopicPartition partition)
Creates the Kafka version specific representation of the given
topic partition.
|
protected void |
emitRecord(T record,
KafkaTopicPartitionState<org.apache.kafka.common.TopicPartition> partition,
long offset,
org.apache.kafka.clients.consumer.ConsumerRecord<?,?> consumerRecord) |
protected String |
getFetcherName()
Gets the name of this fetcher, for thread naming and logging purposes.
|
void |
runFetchLoop() |
addOffsetStateGauge, emitRecord, emitRecordWithTimestamp, emitRecordWithTimestampAndPeriodicWatermark, emitRecordWithTimestampAndPunctuatedWatermark, restoreOffsets, snapshotCurrentState, subscribedPartitions
public Kafka09Fetcher(SourceFunction.SourceContext<T> sourceContext, List<KafkaTopicPartition> assignedPartitions, SerializedValue<AssignerWithPeriodicWatermarks<T>> watermarksPeriodic, SerializedValue<AssignerWithPunctuatedWatermarks<T>> watermarksPunctuated, ProcessingTimeService processingTimeProvider, long autoWatermarkInterval, ClassLoader userCodeClassLoader, boolean enableCheckpointing, String taskNameWithSubtasks, MetricGroup metricGroup, KeyedDeserializationSchema<T> deserializer, Properties kafkaProperties, long pollTimeout, boolean useMetrics) throws Exception
Exception
public void runFetchLoop() throws Exception
runFetchLoop
in class AbstractFetcher<T,org.apache.kafka.common.TopicPartition>
Exception
public void cancel()
cancel
in class AbstractFetcher<T,org.apache.kafka.common.TopicPartition>
protected void emitRecord(T record, KafkaTopicPartitionState<org.apache.kafka.common.TopicPartition> partition, long offset, org.apache.kafka.clients.consumer.ConsumerRecord<?,?> consumerRecord) throws Exception
Exception
protected String getFetcherName()
protected KafkaConsumerCallBridge createCallBridge()
public org.apache.kafka.common.TopicPartition createKafkaPartitionHandle(KafkaTopicPartition partition)
AbstractFetcher
createKafkaPartitionHandle
in class AbstractFetcher<T,org.apache.kafka.common.TopicPartition>
partition
- The Flink representation of the Kafka topic partition.public void commitInternalOffsetsToKafka(Map<KafkaTopicPartition,Long> offsets) throws Exception
AbstractFetcher
commitInternalOffsetsToKafka
in class AbstractFetcher<T,org.apache.kafka.common.TopicPartition>
offsets
- The offsets to commit to Kafka (implementations must increment offsets by 1 before committing).Exception
- This method forwards exceptions.Copyright © 2014–2017 The Apache Software Foundation. All rights reserved.