Modifier and Type | Method and Description |
---|---|
static <T> WatermarkStrategy<T> |
WatermarkStrategy.forBoundedOutOfOrderness(java.time.Duration maxOutOfOrderness)
Creates a watermark strategy for situations where records are out of order, but you can place
an upper bound on how far the events are out of order.
|
static <T> WatermarkStrategy<T> |
WatermarkStrategy.forGenerator(WatermarkGeneratorSupplier<T> generatorSupplier)
Creates a watermark strategy based on an existing
WatermarkGeneratorSupplier . |
static <T> WatermarkStrategy<T> |
WatermarkStrategy.forMonotonousTimestamps()
Creates a watermark strategy for situations with monotonously ascending timestamps.
|
static <T> WatermarkStrategy<T> |
WatermarkStrategy.noWatermarks()
Creates a watermark strategy that generates no watermarks at all.
|
default WatermarkStrategy<T> |
WatermarkStrategy.withIdleness(java.time.Duration idleTimeout)
Creates a new enriched
WatermarkStrategy that also does idleness detection in the
created WatermarkGenerator . |
default WatermarkStrategy<T> |
WatermarkStrategy.withTimestampAssigner(SerializableTimestampAssigner<T> timestampAssigner)
Creates a new
WatermarkStrategy that wraps this strategy but instead uses the given
SerializableTimestampAssigner . |
default WatermarkStrategy<T> |
WatermarkStrategy.withTimestampAssigner(TimestampAssignerSupplier<T> timestampAssigner)
Creates a new
WatermarkStrategy that wraps this strategy but instead uses the given
TimestampAssigner (via a TimestampAssignerSupplier ). |
Modifier and Type | Method and Description |
---|---|
SingleOutputStreamOperator<T> |
DataStream.assignTimestampsAndWatermarks(WatermarkStrategy<T> watermarkStrategy)
Assigns timestamps to the elements in the data stream and generates watermarks to signal
event time progress.
|
Constructor and Description |
---|
DataStreamSource(StreamExecutionEnvironment environment,
Source<T,?,?> source,
WatermarkStrategy<T> watermarkStrategy,
TypeInformation<T> outTypeInfo,
String sourceName)
Constructor for new Sources (FLIP-27).
|
Modifier and Type | Method and Description |
---|---|
<OUT> DataStreamSource<OUT> |
StreamExecutionEnvironment.fromSource(Source<OUT,?,?> source,
WatermarkStrategy<OUT> timestampsAndWatermarks,
String sourceName)
Adds a data
Source to the environment to get a DataStream . |
<OUT> DataStreamSource<OUT> |
StreamExecutionEnvironment.fromSource(Source<OUT,?,?> source,
WatermarkStrategy<OUT> timestampsAndWatermarks,
String sourceName,
TypeInformation<OUT> typeInfo)
Adds a data
Source to the environment to get a DataStream . |
Constructor and Description |
---|
SourceOperator(FunctionWithException<SourceReaderContext,SourceReader<OUT,SplitT>,Exception> readerFactory,
OperatorEventGateway operatorEventGateway,
SimpleVersionedSerializer<SplitT> splitSerializer,
WatermarkStrategy<OUT> watermarkStrategy,
ProcessingTimeService timeService,
Configuration configuration,
String localHostname,
boolean emitProgressiveWatermarks) |
SourceOperatorFactory(Source<OUT,?,?> source,
WatermarkStrategy<OUT> watermarkStrategy) |
SourceOperatorFactory(Source<OUT,?,?> source,
WatermarkStrategy<OUT> watermarkStrategy,
boolean emitProgressiveWatermarks) |
SourceOperatorFactory(Source<OUT,?,?> source,
WatermarkStrategy<OUT> watermarkStrategy,
boolean emitProgressiveWatermarks,
int numCoordinatorWorkerThread) |
Constructor and Description |
---|
PythonTimestampsAndWatermarksOperator(Configuration config,
TypeInformation<IN> inputTypeInfo,
DataStreamPythonFunctionInfo pythonFunctionInfo,
WatermarkStrategy<IN> watermarkStrategy) |
Modifier and Type | Method and Description |
---|---|
static <E> TimestampsAndWatermarks<E> |
TimestampsAndWatermarks.createNoOpEventTimeLogic(WatermarkStrategy<E> watermarkStrategy,
MetricGroup metrics) |
static <E> TimestampsAndWatermarks<E> |
TimestampsAndWatermarks.createProgressiveEventTimeLogic(WatermarkStrategy<E> watermarkStrategy,
MetricGroup metrics,
ProcessingTimeService timeService,
long periodicWatermarkIntervalMillis) |
Modifier and Type | Method and Description |
---|---|
WatermarkStrategy<IN> |
TimestampsAndWatermarksTransformation.getWatermarkStrategy()
Returns the
WatermarkStrategy to use. |
WatermarkStrategy<OUT> |
SourceTransformation.getWatermarkStrategy() |
Constructor and Description |
---|
SourceTransformation(String name,
Source<OUT,SplitT,EnumChkT> source,
WatermarkStrategy<OUT> watermarkStrategy,
TypeInformation<OUT> outputType,
int parallelism)
Creates a new
Transformation with the given name, output type and parallelism. |
TimestampsAndWatermarksTransformation(String name,
int parallelism,
Transformation<IN> input,
WatermarkStrategy<IN> watermarkStrategy)
Creates a new
Transformation with the given name, output type and parallelism. |
Modifier and Type | Method and Description |
---|---|
FlinkKafkaConsumerBase<T> |
FlinkKafkaConsumerBase.assignTimestampsAndWatermarks(WatermarkStrategy<T> watermarkStrategy)
Sets the given
WatermarkStrategy on this consumer. |
Modifier and Type | Method and Description |
---|---|
protected AbstractFetcher<T,?> |
FlinkKafkaConsumer.createFetcher(SourceFunction.SourceContext<T> sourceContext,
Map<KafkaTopicPartition,Long> assignedPartitionsWithInitialOffsets,
SerializedValue<WatermarkStrategy<T>> watermarkStrategy,
StreamingRuntimeContext runtimeContext,
OffsetCommitMode offsetCommitMode,
MetricGroup consumerMetricGroup,
boolean useMetrics) |
protected abstract AbstractFetcher<T,?> |
FlinkKafkaConsumerBase.createFetcher(SourceFunction.SourceContext<T> sourceContext,
Map<KafkaTopicPartition,Long> subscribedPartitionsToStartOffsets,
SerializedValue<WatermarkStrategy<T>> watermarkStrategy,
StreamingRuntimeContext runtimeContext,
OffsetCommitMode offsetCommitMode,
MetricGroup kafkaMetricGroup,
boolean useMetrics)
Creates the fetcher that connect to the Kafka brokers, pulls data, deserialized the data, and
emits it into the data streams.
|
Constructor and Description |
---|
AbstractFetcher(SourceFunction.SourceContext<T> sourceContext,
Map<KafkaTopicPartition,Long> seedPartitionsWithInitialOffsets,
SerializedValue<WatermarkStrategy<T>> watermarkStrategy,
ProcessingTimeService processingTimeProvider,
long autoWatermarkInterval,
ClassLoader userCodeClassLoader,
MetricGroup consumerMetricGroup,
boolean useMetrics) |
KafkaFetcher(SourceFunction.SourceContext<T> sourceContext,
Map<KafkaTopicPartition,Long> assignedPartitionsWithInitialOffsets,
SerializedValue<WatermarkStrategy<T>> watermarkStrategy,
ProcessingTimeService processingTimeProvider,
long autoWatermarkInterval,
ClassLoader userCodeClassLoader,
String taskNameWithSubtasks,
KafkaDeserializationSchema<T> deserializer,
Properties kafkaProperties,
long pollTimeout,
MetricGroup subtaskMetricGroup,
MetricGroup consumerMetricGroup,
boolean useMetrics) |
KafkaShuffleFetcher(SourceFunction.SourceContext<T> sourceContext,
Map<KafkaTopicPartition,Long> assignedPartitionsWithInitialOffsets,
SerializedValue<WatermarkStrategy<T>> watermarkStrategy,
ProcessingTimeService processingTimeProvider,
long autoWatermarkInterval,
ClassLoader userCodeClassLoader,
String taskNameWithSubtasks,
KafkaDeserializationSchema<T> deserializer,
Properties kafkaProperties,
long pollTimeout,
MetricGroup subtaskMetricGroup,
MetricGroup consumerMetricGroup,
boolean useMetrics,
TypeSerializer<T> typeSerializer,
int producerParallelism) |
Modifier and Type | Method and Description |
---|---|
protected AbstractFetcher<T,?> |
FlinkKafkaShuffleConsumer.createFetcher(SourceFunction.SourceContext<T> sourceContext,
Map<KafkaTopicPartition,Long> assignedPartitionsWithInitialOffsets,
SerializedValue<WatermarkStrategy<T>> watermarkStrategy,
StreamingRuntimeContext runtimeContext,
OffsetCommitMode offsetCommitMode,
MetricGroup consumerMetricGroup,
boolean useMetrics) |
Modifier and Type | Field and Description |
---|---|
protected WatermarkStrategy<RowData> |
KafkaDynamicSource.watermarkStrategy
Watermark strategy that is used to generate per-partition watermark.
|
Modifier and Type | Method and Description |
---|---|
void |
KafkaDynamicSource.applyWatermark(WatermarkStrategy<RowData> watermarkStrategy) |
Constructor and Description |
---|
TimestampsAndWatermarksOperator(WatermarkStrategy<T> watermarkStrategy,
boolean emitProgressiveWatermarks) |
Modifier and Type | Class and Description |
---|---|
static class |
AssignerWithPeriodicWatermarksAdapter.Strategy<T>
A WatermarkStrategy that returns an
AssignerWithPeriodicWatermarks wrapped as a
WatermarkGenerator . |
static class |
AssignerWithPunctuatedWatermarksAdapter.Strategy<T>
A WatermarkStrategy that returns an
AssignerWithPunctuatedWatermarks wrapped as a
WatermarkGenerator . |
Modifier and Type | Method and Description |
---|---|
void |
SupportsWatermarkPushDown.applyWatermark(WatermarkStrategy<RowData> watermarkStrategy)
Provides a
WatermarkStrategy which defines how to generate Watermark s in the
stream source. |
Copyright © 2014–2021 The Apache Software Foundation. All rights reserved.