IN
- The input type of the sink.@PublicEvolving public class PulsarSink<IN> extends Object implements TwoPhaseCommittingSink<IN,PulsarCommittable>
PulsarSinkBuilder
to construct a PulsarSink
. The following example shows how to create a PulsarSink receiving records of String
type.
PulsarSink<String> sink = PulsarSink.builder()
.setServiceUrl(operator().serviceUrl())
.setAdminUrl(operator().adminUrl())
.setTopic(topic)
.setSerializationSchema(PulsarSerializationSchema.pulsarSchema(Schema.STRING))
.build();
The sink supports all delivery guarantees described by DeliveryGuarantee
.
DeliveryGuarantee.NONE
does not provide any guarantees: messages may be lost in
case of issues on the Pulsar broker and messages may be duplicated in case of a Flink
failure.
DeliveryGuarantee.AT_LEAST_ONCE
the sink will wait for all outstanding records in
the Pulsar buffers to be acknowledged by the Pulsar producer on a checkpoint. No messages
will be lost in case of any issue with the Pulsar brokers but messages may be duplicated
when Flink restarts.
DeliveryGuarantee.EXACTLY_ONCE
: In this mode the PulsarSink will write all messages
in a Pulsar transaction that will be committed to Pulsar on a checkpoint. Thus, no
duplicates will be seen in case of a Flink restart. However, this delays record writing
effectively until a checkpoint is written, so adjust the checkpoint duration accordingly.
Additionally, it is highly recommended to tweak Pulsar transaction timeout (link) >>
maximum checkpoint duration + maximum restart duration or data loss may happen when Pulsar
expires an uncommitted transaction.
See PulsarSinkBuilder
for more details.
TwoPhaseCommittingSink.PrecommittingSinkWriter<InputT,CommT>
Sink.InitContext
Modifier and Type | Method and Description |
---|---|
static <IN> PulsarSinkBuilder<IN> |
builder()
Create a
PulsarSinkBuilder to construct a new PulsarSink . |
Committer<PulsarCommittable> |
createCommitter()
Creates a
Committer that permanently makes the previously written data visible
through Committer.commit(Collection) . |
TwoPhaseCommittingSink.PrecommittingSinkWriter<IN,PulsarCommittable> |
createWriter(Sink.InitContext initContext)
Creates a
TwoPhaseCommittingSink.PrecommittingSinkWriter that creates committables on checkpoint or end of
input. |
SimpleVersionedSerializer<PulsarCommittable> |
getCommittableSerializer()
Returns the serializer of the committable type.
|
public static <IN> PulsarSinkBuilder<IN> builder()
PulsarSinkBuilder
to construct a new PulsarSink
.IN
- Type of incoming records.@Internal public TwoPhaseCommittingSink.PrecommittingSinkWriter<IN,PulsarCommittable> createWriter(Sink.InitContext initContext)
TwoPhaseCommittingSink
TwoPhaseCommittingSink.PrecommittingSinkWriter
that creates committables on checkpoint or end of
input.createWriter
in interface Sink<IN>
createWriter
in interface TwoPhaseCommittingSink<IN,PulsarCommittable>
initContext
- the runtime context.@Internal public Committer<PulsarCommittable> createCommitter()
TwoPhaseCommittingSink
Committer
that permanently makes the previously written data visible
through Committer.commit(Collection)
.createCommitter
in interface TwoPhaseCommittingSink<IN,PulsarCommittable>
@Internal public SimpleVersionedSerializer<PulsarCommittable> getCommittableSerializer()
TwoPhaseCommittingSink
getCommittableSerializer
in interface TwoPhaseCommittingSink<IN,PulsarCommittable>
Copyright © 2014–2023 The Apache Software Foundation. All rights reserved.