@Internal public class KafkaSourceReader<T> extends SingleThreadMultiplexSourceReaderBase<org.apache.kafka.clients.consumer.ConsumerRecord<byte[],byte[]>,T,KafkaPartitionSplit,KafkaPartitionSplitState>
config, context, options, recordEmitter, splitFetcherManager
Constructor and Description |
---|
KafkaSourceReader(FutureCompletingBlockingQueue<RecordsWithSplitIds<org.apache.kafka.clients.consumer.ConsumerRecord<byte[],byte[]>>> elementsQueue,
KafkaSourceFetcherManager kafkaSourceFetcherManager,
RecordEmitter<org.apache.kafka.clients.consumer.ConsumerRecord<byte[],byte[]>,T,KafkaPartitionSplitState> recordEmitter,
Configuration config,
SourceReaderContext context,
KafkaSourceReaderMetrics kafkaSourceReaderMetrics) |
Modifier and Type | Method and Description |
---|---|
protected KafkaPartitionSplitState |
initializedState(KafkaPartitionSplit split)
When new splits are added to the reader.
|
void |
notifyCheckpointComplete(long checkpointId)
We have an empty default implementation here because most source readers do not have to
implement the method.
|
protected void |
onSplitFinished(Map<String,KafkaPartitionSplitState> finishedSplitIds)
Handles the finished splits to clean the state if needed.
|
void |
pauseOrResumeSplits(Collection<String> splitsToPause,
Collection<String> splitsToResume)
Pauses or resumes reading of individual source splits.
|
List<KafkaPartitionSplit> |
snapshotState(long checkpointId)
Checkpoint on the state of the source.
|
protected KafkaPartitionSplit |
toSplitType(String splitId,
KafkaPartitionSplitState splitState)
Convert a mutable SplitStateT to immutable SplitT.
|
addSplits, close, getNumberOfCurrentlyAssignedSplits, handleSourceEvents, isAvailable, notifyNoMoreSplits, pollNext, start
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
notifyCheckpointAborted
public KafkaSourceReader(FutureCompletingBlockingQueue<RecordsWithSplitIds<org.apache.kafka.clients.consumer.ConsumerRecord<byte[],byte[]>>> elementsQueue, KafkaSourceFetcherManager kafkaSourceFetcherManager, RecordEmitter<org.apache.kafka.clients.consumer.ConsumerRecord<byte[],byte[]>,T,KafkaPartitionSplitState> recordEmitter, Configuration config, SourceReaderContext context, KafkaSourceReaderMetrics kafkaSourceReaderMetrics)
protected void onSplitFinished(Map<String,KafkaPartitionSplitState> finishedSplitIds)
SourceReaderBase
onSplitFinished
in class SourceReaderBase<org.apache.kafka.clients.consumer.ConsumerRecord<byte[],byte[]>,T,KafkaPartitionSplit,KafkaPartitionSplitState>
public List<KafkaPartitionSplit> snapshotState(long checkpointId)
SourceReader
snapshotState
in interface SourceReader<T,KafkaPartitionSplit>
snapshotState
in class SourceReaderBase<org.apache.kafka.clients.consumer.ConsumerRecord<byte[],byte[]>,T,KafkaPartitionSplit,KafkaPartitionSplitState>
public void notifyCheckpointComplete(long checkpointId) throws Exception
SourceReader
checkpointId
- The ID of the checkpoint that has been completed.Exception
- This method can propagate exceptions, which leads to a failure/recovery for
the task. Note that this will NOT lead to the checkpoint being revoked.CheckpointListener.notifyCheckpointComplete(long)
protected KafkaPartitionSplitState initializedState(KafkaPartitionSplit split)
SourceReaderBase
initializedState
in class SourceReaderBase<org.apache.kafka.clients.consumer.ConsumerRecord<byte[],byte[]>,T,KafkaPartitionSplit,KafkaPartitionSplitState>
split
- a newly added split.protected KafkaPartitionSplit toSplitType(String splitId, KafkaPartitionSplitState splitState)
SourceReaderBase
toSplitType
in class SourceReaderBase<org.apache.kafka.clients.consumer.ConsumerRecord<byte[],byte[]>,T,KafkaPartitionSplit,KafkaPartitionSplitState>
splitState
- splitState.public void pauseOrResumeSplits(Collection<String> splitsToPause, Collection<String> splitsToResume)
SourceReader
Note that no other methods can be called in parallel, so updating subscriptions can be done atomically. This method is simply providing connectors with more expressive APIs the opportunity to update all subscriptions at once.
This is currently used to align the watermarks of splits, if watermark alignment is used and the source reads from more than one split.
The default implementation throws an UnsupportedOperationException
where the
default implementation will be removed in future releases. To be compatible with future
releases, it is recommended to implement this method and override the default implementation.
pauseOrResumeSplits
in interface SourceReader<T,KafkaPartitionSplit>
pauseOrResumeSplits
in class SourceReaderBase<org.apache.kafka.clients.consumer.ConsumerRecord<byte[],byte[]>,T,KafkaPartitionSplit,KafkaPartitionSplitState>
splitsToPause
- the splits to pausesplitsToResume
- the splits to resumeCopyright © 2014–2023 The Apache Software Foundation. All rights reserved.