T
- Type of the elements handled by this sink@PublicEvolving public class ElasticsearchSink<T> extends ElasticsearchSinkBase<T,org.elasticsearch.client.Client>
ActionRequests
against a cluster for each incoming element.
When using the first constructor ElasticsearchSink(java.util.Map, ElasticsearchSinkFunction)
the sink will create a local Node
for communicating with the Elasticsearch cluster. When using the second
constructor ElasticsearchSink(java.util.Map, java.util.List, ElasticsearchSinkFunction)
a
TransportClient
will be used instead.
Attention: When using the TransportClient
the sink will fail if no cluster
can be connected to. When using the local Node
for communicating, the sink will block and wait for a cluster
to come online.
The Map
passed to the constructor is used to create the Node
or TransportClient
. The config
keys can be found in the Elasticsearch documentation. An important setting is
cluster.name
, which should be set to the name of the cluster that the sink should emit to.
Internally, the sink will use a BulkProcessor
to send ActionRequests
.
This will buffer elements before sending a request to the cluster. The behaviour of the
BulkProcessor
can be configured using these config keys:
bulk.flush.max.actions
: Maximum amount of elements to buffer
bulk.flush.max.size.mb
: Maximum amount of data (in megabytes) to buffer
bulk.flush.interval.ms
: Interval at which to flush data regardless of the other two
settings in milliseconds
You also have to provide an ElasticsearchSinkFunction
. This is used to create multiple
ActionRequests
for each incoming element. See the class level documentation of
ElasticsearchSinkFunction
for an example.
ElasticsearchSinkBase.BulkFlushBackoffPolicy, ElasticsearchSinkBase.FlushBackoffType
SinkFunction.Context<T>
CONFIG_KEY_BULK_FLUSH_BACKOFF_DELAY, CONFIG_KEY_BULK_FLUSH_BACKOFF_ENABLE, CONFIG_KEY_BULK_FLUSH_BACKOFF_RETRIES, CONFIG_KEY_BULK_FLUSH_BACKOFF_TYPE, CONFIG_KEY_BULK_FLUSH_INTERVAL_MS, CONFIG_KEY_BULK_FLUSH_MAX_ACTIONS, CONFIG_KEY_BULK_FLUSH_MAX_SIZE_MB
Constructor and Description |
---|
ElasticsearchSink(Map<String,String> userConfig,
ElasticsearchSinkFunction<T> elasticsearchSinkFunction)
Creates a new
ElasticsearchSink that connects to the cluster using an embedded Node . |
ElasticsearchSink(Map<String,String> userConfig,
ElasticsearchSinkFunction<T> elasticsearchSinkFunction,
ActionRequestFailureHandler failureHandler)
Creates a new
ElasticsearchSink that connects to the cluster using an embedded Node . |
ElasticsearchSink(Map<String,String> userConfig,
IndexRequestBuilder<T> indexRequestBuilder)
Deprecated.
Deprecated since version 1.2, to be removed at version 2.0.
Please use
ElasticsearchSink(Map, ElasticsearchSinkFunction) instead. |
ElasticsearchSink(Map<String,String> userConfig,
List<org.elasticsearch.common.transport.TransportAddress> transportAddresses,
ElasticsearchSinkFunction<T> elasticsearchSinkFunction)
Creates a new
ElasticsearchSink that connects to the cluster using a TransportClient . |
ElasticsearchSink(Map<String,String> userConfig,
List<org.elasticsearch.common.transport.TransportAddress> transportAddresses,
ElasticsearchSinkFunction<T> elasticsearchSinkFunction,
ActionRequestFailureHandler failureHandler)
Creates a new
ElasticsearchSink that connects to the cluster using a TransportClient . |
ElasticsearchSink(Map<String,String> userConfig,
List<org.elasticsearch.common.transport.TransportAddress> transportAddresses,
IndexRequestBuilder<T> indexRequestBuilder)
Deprecated.
Deprecated since 1.2, to be removed at 2.0.
Please use
ElasticsearchSink(Map, List, ElasticsearchSinkFunction) instead. |
buildBulkProcessor, close, disableFlushOnCheckpoint, initializeState, invoke, open, snapshotState
getIterationRuntimeContext, getRuntimeContext, setRuntimeContext
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
invoke
@Deprecated public ElasticsearchSink(Map<String,String> userConfig, IndexRequestBuilder<T> indexRequestBuilder)
ElasticsearchSink(Map, ElasticsearchSinkFunction)
instead.ElasticsearchSink
that connects to the cluster using an embedded Node
.userConfig
- The map of user settings that are used when constructing the Node
and BulkProcessor
indexRequestBuilder
- This is used to generate the IndexRequest from the incoming element@Deprecated public ElasticsearchSink(Map<String,String> userConfig, List<org.elasticsearch.common.transport.TransportAddress> transportAddresses, IndexRequestBuilder<T> indexRequestBuilder)
ElasticsearchSink(Map, List, ElasticsearchSinkFunction)
instead.ElasticsearchSink
that connects to the cluster using a TransportClient
.userConfig
- The map of user settings that are used when constructing the TransportClient
and BulkProcessor
transportAddresses
- The addresses of Elasticsearch nodes to which to connect using a TransportClient
indexRequestBuilder
- This is used to generate a IndexRequest
from the incoming elementpublic ElasticsearchSink(Map<String,String> userConfig, ElasticsearchSinkFunction<T> elasticsearchSinkFunction)
ElasticsearchSink
that connects to the cluster using an embedded Node
.userConfig
- The map of user settings that are used when constructing the embedded Node
and BulkProcessor
elasticsearchSinkFunction
- This is used to generate multiple ActionRequest
from the incoming elementpublic ElasticsearchSink(Map<String,String> userConfig, List<org.elasticsearch.common.transport.TransportAddress> transportAddresses, ElasticsearchSinkFunction<T> elasticsearchSinkFunction)
ElasticsearchSink
that connects to the cluster using a TransportClient
.userConfig
- The map of user settings that are used when constructing the TransportClient
and BulkProcessor
transportAddresses
- The addresses of Elasticsearch nodes to which to connect using a TransportClient
elasticsearchSinkFunction
- This is used to generate multiple ActionRequest
from the incoming elementpublic ElasticsearchSink(Map<String,String> userConfig, ElasticsearchSinkFunction<T> elasticsearchSinkFunction, ActionRequestFailureHandler failureHandler)
ElasticsearchSink
that connects to the cluster using an embedded Node
.userConfig
- The map of user settings that are used when constructing the embedded Node
and BulkProcessor
elasticsearchSinkFunction
- This is used to generate multiple ActionRequest
from the incoming elementfailureHandler
- This is used to handle failed ActionRequest
public ElasticsearchSink(Map<String,String> userConfig, List<org.elasticsearch.common.transport.TransportAddress> transportAddresses, ElasticsearchSinkFunction<T> elasticsearchSinkFunction, ActionRequestFailureHandler failureHandler)
ElasticsearchSink
that connects to the cluster using a TransportClient
.userConfig
- The map of user settings that are used when constructing the TransportClient
and BulkProcessor
transportAddresses
- The addresses of Elasticsearch nodes to which to connect using a TransportClient
elasticsearchSinkFunction
- This is used to generate multiple ActionRequest
from the incoming elementfailureHandler
- This is used to handle failed ActionRequest
Copyright © 2014–2020 The Apache Software Foundation. All rights reserved.