T
- The type of the events/records produced by this source.@Internal public class TestFileSource<T> extends AbstractFileSource<T,FileSourceSplit> implements DynamicParallelismInference
FileSource
is a final class, so we can't extend it directly.Modifier and Type | Class and Description |
---|---|
static class |
TestFileSource.TestFileSourceBuilder<T>
The builder for the
FileSource , to configure the various behaviors. |
AbstractFileSource.AbstractFileSourceBuilder<T,SplitT extends FileSourceSplit,SELF extends AbstractFileSource.AbstractFileSourceBuilder<T,SplitT,SELF>>
DynamicParallelismInference.Context
Modifier and Type | Field and Description |
---|---|
static FileEnumerator.Provider |
DEFAULT_NON_SPLITTABLE_FILE_ENUMERATOR
The default file enumerator used for non-splittable formats.
|
static FileSplitAssigner.Provider |
DEFAULT_SPLIT_ASSIGNER
The default split assigner, a lazy locality-aware assigner.
|
static FileEnumerator.Provider |
DEFAULT_SPLITTABLE_FILE_ENUMERATOR
The default file enumerator used for splittable formats.
|
Modifier and Type | Method and Description |
---|---|
static <T> TestFileSource.TestFileSourceBuilder<T> |
forBulkFileFormat(BulkFormat<T,FileSourceSplit> bulkFormat,
Path... paths)
Builds a new
FileSource using a BulkFormat to read batches of records from
files. |
Boundedness |
getBoundedness()
Get the boundedness of this source.
|
SimpleVersionedSerializer<FileSourceSplit> |
getSplitSerializer()
Creates a serializer for the source splits.
|
int |
inferParallelism(DynamicParallelismInference.Context dynamicParallelismContext)
The method is invoked on the master (JobManager) before the initialization of the source
vertex.
|
createEnumerator, createReader, getAssignerFactory, getContinuousEnumerationSettings, getEnumeratorCheckpointSerializer, getProducedType, restoreEnumerator
public static final FileSplitAssigner.Provider DEFAULT_SPLIT_ASSIGNER
public static final FileEnumerator.Provider DEFAULT_SPLITTABLE_FILE_ENUMERATOR
public static final FileEnumerator.Provider DEFAULT_NON_SPLITTABLE_FILE_ENUMERATOR
public SimpleVersionedSerializer<FileSourceSplit> getSplitSerializer()
Source
getSplitSerializer
in interface Source<T,FileSourceSplit,PendingSplitsCheckpoint<FileSourceSplit>>
getSplitSerializer
in class AbstractFileSource<T,FileSourceSplit>
public Boundedness getBoundedness()
Source
getBoundedness
in interface Source<T,FileSourceSplit,PendingSplitsCheckpoint<FileSourceSplit>>
getBoundedness
in class AbstractFileSource<T,FileSourceSplit>
public int inferParallelism(DynamicParallelismInference.Context dynamicParallelismContext)
DynamicParallelismInference
inferParallelism
in interface DynamicParallelismInference
dynamicParallelismContext
- The context to get dynamic parallelism decision infos.public static <T> TestFileSource.TestFileSourceBuilder<T> forBulkFileFormat(BulkFormat<T,FileSourceSplit> bulkFormat, Path... paths)
FileSource
using a BulkFormat
to read batches of records from
files.
Examples for bulk readers are compressed and vectorized formats such as ORC or Parquet.
Copyright © 2014–2024 The Apache Software Foundation. All rights reserved.