IN
- Type of the messages to write into Kafka.public abstract class FlinkKafkaProducerBase<IN> extends RichSinkFunction<IN> implements Checkpointed<Serializable>
Modifier and Type | Field and Description |
---|---|
protected Exception |
asyncException
Errors encountered in the async producer are stored here
|
protected org.apache.kafka.clients.producer.Callback |
callback
The callback than handles error propagation or logging callbacks
|
protected String |
defaultTopicId
The name of the default topic this producer is writing data to
|
protected boolean |
flushOnCheckpoint
If true, the producer will wait until all outstanding records have been send to the broker.
|
static String |
KEY_DISABLE_METRICS
Configuration key for disabling the metrics reporting
|
protected boolean |
logFailuresOnly
Flag indicating whether to accept failures (and log them), or to fail on failures
|
protected KafkaPartitioner<IN> |
partitioner
User-provided partitioner for assigning an object to a Kafka partition.
|
protected int[] |
partitions
Array with the partition ids of the given defaultTopicId
The size of this array is the number of partitions
|
protected long |
pendingRecords
Number of unacknowledged records.
|
protected SerializableObject |
pendingRecordsLock
Lock for accessing the pending records
|
protected org.apache.kafka.clients.producer.KafkaProducer<byte[],byte[]> |
producer
KafkaProducer instance
|
protected Properties |
producerConfig
User defined properties for the Producer
|
protected KeyedSerializationSchema<IN> |
schema
(Serializable) SerializationSchema for turning objects used with Flink into
byte[] for Kafka.
|
Constructor and Description |
---|
FlinkKafkaProducerBase(String defaultTopicId,
KeyedSerializationSchema<IN> serializationSchema,
Properties producerConfig,
KafkaPartitioner<IN> customPartitioner)
The main constructor for creating a FlinkKafkaProducer.
|
Modifier and Type | Method and Description |
---|---|
protected void |
checkErroneous() |
void |
close()
Tear-down method for the user code.
|
protected abstract void |
flush()
Flush pending records.
|
protected <K,V> org.apache.kafka.clients.producer.KafkaProducer<K,V> |
getKafkaProducer(Properties props)
Used for testing only
|
static Properties |
getPropertiesFromBrokerList(String brokerList) |
void |
invoke(IN next)
Called when new data arrives to the sink, and forwards it to Kafka.
|
protected long |
numPendingRecords() |
void |
open(Configuration configuration)
Initializes the connection to Kafka.
|
void |
restoreState(Serializable state)
Restores the state of the function or operator to that of a previous checkpoint.
|
void |
setFlushOnCheckpoint(boolean flush)
If set to true, the Flink producer will wait for all outstanding messages in the Kafka buffers
to be acknowledged by the Kafka producer on a checkpoint.
|
void |
setLogFailuresOnly(boolean logFailuresOnly)
Defines whether the producer should fail on errors, or only log them.
|
Serializable |
snapshotState(long checkpointId,
long checkpointTimestamp)
Gets the current state of the function of operator.
|
getIterationRuntimeContext, getRuntimeContext, setRuntimeContext
public static final String KEY_DISABLE_METRICS
protected final int[] partitions
protected final Properties producerConfig
protected final String defaultTopicId
protected final KeyedSerializationSchema<IN> schema
protected final KafkaPartitioner<IN> partitioner
protected boolean logFailuresOnly
protected boolean flushOnCheckpoint
protected transient org.apache.kafka.clients.producer.KafkaProducer<byte[],byte[]> producer
protected transient org.apache.kafka.clients.producer.Callback callback
protected transient volatile Exception asyncException
protected final SerializableObject pendingRecordsLock
protected long pendingRecords
public FlinkKafkaProducerBase(String defaultTopicId, KeyedSerializationSchema<IN> serializationSchema, Properties producerConfig, KafkaPartitioner<IN> customPartitioner)
defaultTopicId
- The default topic to write data toserializationSchema
- A serializable serialization schema for turning user objects into a kafka-consumable byte[] supporting key/value messagesproducerConfig
- Configuration properties for the KafkaProducer. 'bootstrap.servers.' is the only required argument.customPartitioner
- A serializable partitioner for assigning messages to Kafka partitions. Passing null will use Kafka's partitionerpublic void setLogFailuresOnly(boolean logFailuresOnly)
logFailuresOnly
- The flag to indicate logging-only on exceptions.public void setFlushOnCheckpoint(boolean flush)
flush
- Flag indicating the flushing mode (true = flush on checkpoint)protected <K,V> org.apache.kafka.clients.producer.KafkaProducer<K,V> getKafkaProducer(Properties props)
public void open(Configuration configuration)
open
in interface RichFunction
open
in class AbstractRichFunction
configuration
- The configuration containing the parameters attached to the contract.Configuration
public void invoke(IN next) throws Exception
invoke
in interface SinkFunction<IN>
invoke
in class RichSinkFunction<IN>
next
- The incoming dataException
public void close() throws Exception
RichFunction
This method can be used for clean up work.
close
in interface RichFunction
close
in class AbstractRichFunction
Exception
- Implementations may forward exceptions, which are caught by the runtime. When the
runtime catches an exception, it aborts the task and lets the fail-over logic
decide whether to retry the task execution.protected abstract void flush()
public Serializable snapshotState(long checkpointId, long checkpointTimestamp) throws Exception
Checkpointed
snapshotState
in interface Checkpointed<Serializable>
checkpointId
- The ID of the checkpoint.checkpointTimestamp
- The timestamp of the checkpoint, as derived by
System.currentTimeMillis() on the JobManager.Exception
- Thrown if the creation of the state object failed. This causes the
checkpoint to fail. The system may decide to fail the operation (and trigger
recovery), or to discard this checkpoint attempt and to continue running
and to try again with the next checkpoint attempt.public void restoreState(Serializable state)
Checkpointed
restoreState
in interface Checkpointed<Serializable>
state
- The state to be restored.public static Properties getPropertiesFromBrokerList(String brokerList)
protected long numPendingRecords()
Copyright © 2014–2017 The Apache Software Foundation. All rights reserved.