IN
- The type of input elements.@Internal public class FileWriter<IN> extends Object implements SinkWriter<IN,FileSinkCommittable,FileWriterBucketState>, Sink.ProcessingTimeService.ProcessingTimeCallback
SinkWriter
implementation for FileSink
.
It writes data to and manages the different active buckes
in the
FileSink
.
SinkWriter.Context
Constructor and Description |
---|
FileWriter(Path basePath,
BucketAssigner<IN,String> bucketAssigner,
FileWriterBucketFactory<IN> bucketFactory,
BucketWriter<IN,String> bucketWriter,
RollingPolicy<IN,String> rollingPolicy,
OutputFileConfig outputFileConfig,
Sink.ProcessingTimeService processingTimeService,
long bucketCheckInterval)
A constructor creating a new empty bucket manager.
|
Modifier and Type | Method and Description |
---|---|
void |
close() |
void |
initializeState(List<FileWriterBucketState> bucketStates)
Initializes the state after recovery from a failure.
|
void |
onProcessingTime(long time)
This method is invoked with the time which the callback register for.
|
List<FileSinkCommittable> |
prepareCommit(boolean flush)
Prepare for a commit.
|
List<FileWriterBucketState> |
snapshotState() |
void |
write(IN element,
SinkWriter.Context context)
Add an element to the writer.
|
public FileWriter(Path basePath, BucketAssigner<IN,String> bucketAssigner, FileWriterBucketFactory<IN> bucketFactory, BucketWriter<IN,String> bucketWriter, RollingPolicy<IN,String> rollingPolicy, OutputFileConfig outputFileConfig, Sink.ProcessingTimeService processingTimeService, long bucketCheckInterval)
basePath
- The base path for our buckets.bucketAssigner
- The BucketAssigner
provided by the user.bucketFactory
- The FileWriterBucketFactory
to be used to create buckets.bucketWriter
- The BucketWriter
to be used when writing data.rollingPolicy
- The RollingPolicy
as specified by the user.public void initializeState(List<FileWriterBucketState> bucketStates) throws IOException
During this process:
bucketStates
- the state holding recovered state about active buckets.IOException
- if anything goes wrong during retrieving the state or
restoring/committing of any in-progress/pending part filespublic void write(IN element, SinkWriter.Context context) throws IOException
SinkWriter
write
in interface SinkWriter<IN,FileSinkCommittable,FileWriterBucketState>
element
- The input recordcontext
- The additional information about the input recordIOException
- if fail to add an element.public List<FileSinkCommittable> prepareCommit(boolean flush) throws IOException
SinkWriter
This will be called before we checkpoint the Writer's state in Streaming execution mode.
prepareCommit
in interface SinkWriter<IN,FileSinkCommittable,FileWriterBucketState>
flush
- Whether flushing the un-staged data or notIOException
- if fail to prepare for a commit.public List<FileWriterBucketState> snapshotState() throws IOException
snapshotState
in interface SinkWriter<IN,FileSinkCommittable,FileWriterBucketState>
IOException
- if fail to snapshot writer's state.public void close()
close
in interface AutoCloseable
public void onProcessingTime(long time) throws IOException
Sink.ProcessingTimeService.ProcessingTimeCallback
onProcessingTime
in interface Sink.ProcessingTimeService.ProcessingTimeCallback
time
- The time this callback was registered for.IOException
Copyright © 2014–2021 The Apache Software Foundation. All rights reserved.