@Deprecated public class AvroKeyValueSinkWriter<K,V> extends StreamWriterBase<Tuple2<K,V>> implements Writer<Tuple2<K,V>>, InputTypeConfigurable
Usage:
BucketingSink<Tuple2<Long, Long>> sink = new BucketingSink<Tuple2<Long, Long>>("/tmp/path");
sink.setBucketer(new DateTimeBucketer<Tuple2<Long, Long>>("yyyy-MM-dd/HH/mm/"));
sink.setPendingSuffix(".avro");
Map<String, String> properties = new HashMap<>();
Schema longSchema = Schema.create(Type.LONG);
String keySchema = longSchema.toString();
String valueSchema = longSchema.toString();
properties.put(AvroKeyValueSinkWriter.CONF_OUTPUT_KEY_SCHEMA, keySchema);
properties.put(AvroKeyValueSinkWriter.CONF_OUTPUT_VALUE_SCHEMA, valueSchema);
properties.put(AvroKeyValueSinkWriter.CONF_COMPRESS, Boolean.toString(true));
properties.put(AvroKeyValueSinkWriter.CONF_COMPRESS_CODEC, DataFileConstants.SNAPPY_CODEC);
sink.setWriter(new AvroKeyValueSinkWriter<Long, Long>(properties));
sink.setBatchSize(1024 * 1024 * 64); // this is 64 MB,
Modifier and Type | Class and Description |
---|---|
static class |
AvroKeyValueSinkWriter.AvroKeyValue<K,V>
Deprecated.
A reusable Avro generic record for writing key/value pairs to the file.
|
Modifier and Type | Field and Description |
---|---|
static String |
CONF_COMPRESS
Deprecated.
|
static String |
CONF_COMPRESS_CODEC
Deprecated.
|
static String |
CONF_DEFLATE_LEVEL
Deprecated.
|
static String |
CONF_OUTPUT_KEY_SCHEMA
Deprecated.
|
static String |
CONF_OUTPUT_VALUE_SCHEMA
Deprecated.
|
static String |
CONF_XZ_LEVEL
Deprecated.
|
Modifier | Constructor and Description |
---|---|
protected |
AvroKeyValueSinkWriter(AvroKeyValueSinkWriter<K,V> other)
Deprecated.
|
|
AvroKeyValueSinkWriter(Map<String,String> properties)
Deprecated.
C'tor for the writer.
|
Modifier and Type | Method and Description |
---|---|
void |
close()
Deprecated.
Closes the
Writer . |
AvroKeyValueSinkWriter<K,V> |
duplicate()
Deprecated.
Duplicates the
Writer . |
long |
flush()
Deprecated.
Flushes out any internally held data, and returns the offset that the file must be truncated
to at recovery.
|
void |
open(org.apache.hadoop.fs.FileSystem fs,
org.apache.hadoop.fs.Path path)
Deprecated.
Initializes the
Writer for a newly opened bucket file. |
void |
setInputType(TypeInformation<?> type,
ExecutionConfig executionConfig)
Deprecated.
Method that is called on an
OutputFormat when it is
passed to the DataSet's output method. |
void |
write(Tuple2<K,V> element)
Deprecated.
Writes one element to the bucket file.
|
getPos, getStream, isSyncOnFlush, setSyncOnFlush
public static final String CONF_OUTPUT_KEY_SCHEMA
public static final String CONF_OUTPUT_VALUE_SCHEMA
public static final String CONF_COMPRESS
public static final String CONF_COMPRESS_CODEC
public static final String CONF_DEFLATE_LEVEL
public static final String CONF_XZ_LEVEL
public AvroKeyValueSinkWriter(Map<String,String> properties)
You can provide different properties that will be used to configure avro key-value writer as simple properties map(see example above)
properties
- protected AvroKeyValueSinkWriter(AvroKeyValueSinkWriter<K,V> other)
public void open(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path path) throws IOException
Writer
Writer
for a newly opened bucket file. Any internal per-bucket
initialization should be performed here.public void close() throws IOException
Writer
Writer
. If the writer is already closed, no action will be taken. The call
should close all state related to the current output file, including the output stream opened
in open
.public long flush() throws IOException
Writer
public void write(Tuple2<K,V> element) throws IOException
Writer
public void setInputType(TypeInformation<?> type, ExecutionConfig executionConfig)
InputTypeConfigurable
OutputFormat
when it is
passed to the DataSet's output method. May be used to configures the output format based on
the data type.setInputType
in interface InputTypeConfigurable
type
- The data type of the input.executionConfig
- The execution config for this parallel execution.Copyright © 2014–2021 The Apache Software Foundation. All rights reserved.