@PublicEvolving public interface SupportsWatermarkPushDown
ScanTableSource
.
The concept of watermarks defines when time operations based on an event time attribute will be triggered. A watermark tells operators that no elements with a timestamp older or equal to the watermark timestamp should arrive at the operator. Thus, watermarks are a trade-off between latency and completeness.
Given the following SQL:
CREATE TABLE t (i INT, ts TIMESTAMP(3), WATERMARK FOR ts AS ts - INTERVAL '5' SECOND) // `ts` becomes a time attribute
In the above example, generated watermarks are lagging 5 seconds behind the highest seen timestamp.
By default, if this interface is not implemented, watermarks are generated in a subsequent operation after the source.
However, for correctness, it might be necessary to perform the watermark generation as early as possible in order to be close to the actual data generation within a source's data partition.
This interface provides a SupportsWatermarkPushDown.WatermarkProvider
that needs to be applied to a runtime
implementation. Most built-in Flink sources provide a way of setting the watermark generator.
Note: In many cases, watermarks are generated from computed columns. If a source implements
this interface, it is recommended to also implement SupportsComputedColumnPushDown
.
Modifier and Type | Interface and Description |
---|---|
static interface |
SupportsWatermarkPushDown.WatermarkProvider
Provides actual runtime implementation for generating watermarks.
|
Modifier and Type | Method and Description |
---|---|
void |
applyWatermark(SupportsWatermarkPushDown.WatermarkProvider provider)
Provides actual runtime implementation for generating watermarks.
|
void applyWatermark(SupportsWatermarkPushDown.WatermarkProvider provider)
There exist different interfaces for runtime implementation which is why SupportsWatermarkPushDown.WatermarkProvider
serves as the base interface. Concrete SupportsWatermarkPushDown.WatermarkProvider
interfaces might be located in other Flink modules.
See org.apache.flink.table.connector.source.abilities
in flink-table-api-java-bridge
.
Implementations need to perform an instanceof
check and fail with an exception if
the given SupportsWatermarkPushDown.WatermarkProvider
is unsupported.
Copyright © 2014–2021 The Apache Software Foundation. All rights reserved.