Constructor and Description |
---|
DistributedCache(Map<String,Future<Path>> cacheCopyTasks) |
Constructor and Description |
---|
AbstractRuntimeUDFContext(TaskInfo taskInfo,
ClassLoader userCodeClassLoader,
ExecutionConfig executionConfig,
Map<String,Accumulator<?,?>> accumulators,
Map<String,Future<Path>> cpTasks,
MetricGroup metrics) |
RuntimeUDFContext(TaskInfo taskInfo,
ClassLoader userCodeClassLoader,
ExecutionConfig executionConfig,
Map<String,Future<Path>> cpTasks,
Map<String,Accumulator<?,?>> accumulators,
MetricGroup metrics) |
Modifier and Type | Field and Description |
---|---|
protected Path |
FileInputFormat.filePath
The path to the file that contains the input.
|
protected Path |
FileOutputFormat.outputFilePath
The path of the file to be written.
|
Modifier and Type | Method and Description |
---|---|
Path |
FileInputFormat.getFilePath() |
Path |
FileOutputFormat.getOutputFilePath() |
Modifier and Type | Method and Description |
---|---|
boolean |
GlobFilePathFilter.filterPath(Path filePath) |
abstract boolean |
FilePathFilter.filterPath(Path filePath)
Returns
true if the filePath given is to be
ignored when processing a directory, e.g. |
boolean |
FilePathFilter.DefaultFilter.filterPath(Path filePath) |
protected FileInputFormat.FileBaseStatistics |
FileInputFormat.getFileStats(FileInputFormat.FileBaseStatistics cachedStats,
Path filePath,
FileSystem fs,
ArrayList<FileStatus> files) |
void |
FileInputFormat.setFilePath(Path filePath) |
void |
FileOutputFormat.setOutputFilePath(Path path) |
Constructor and Description |
---|
DelimitedInputFormat(Path filePath,
Configuration configuration) |
FileInputFormat(Path filePath) |
FileOutputFormat(Path outputPath) |
GenericCsvInputFormat(Path filePath) |
Modifier and Type | Method and Description |
---|---|
Path |
CsvReader.getFilePath() |
Constructor and Description |
---|
CsvInputFormat(Path filePath) |
CsvOutputFormat(Path outputPath)
Creates an instance of CsvOutputFormat.
|
CsvOutputFormat(Path outputPath,
String fieldDelimiter)
Creates an instance of CsvOutputFormat.
|
CsvOutputFormat(Path outputPath,
String recordDelimiter,
String fieldDelimiter)
Creates an instance of CsvOutputFormat.
|
CsvReader(Path filePath,
ExecutionEnvironment executionContext) |
PojoCsvInputFormat(Path filePath,
PojoTypeInfo<OUT> pojoTypeInfo) |
PojoCsvInputFormat(Path filePath,
PojoTypeInfo<OUT> pojoTypeInfo,
boolean[] includedFieldsMask) |
PojoCsvInputFormat(Path filePath,
PojoTypeInfo<OUT> pojoTypeInfo,
int[] includedFieldsMask) |
PojoCsvInputFormat(Path filePath,
PojoTypeInfo<OUT> pojoTypeInfo,
String[] fieldNames) |
PojoCsvInputFormat(Path filePath,
PojoTypeInfo<OUT> pojoTypeInfo,
String[] fieldNames,
boolean[] includedFieldsMask) |
PojoCsvInputFormat(Path filePath,
PojoTypeInfo<OUT> pojoTypeInfo,
String[] fieldNames,
int[] includedFieldsMask) |
PojoCsvInputFormat(Path filePath,
String lineDelimiter,
String fieldDelimiter,
PojoTypeInfo<OUT> pojoTypeInfo) |
PojoCsvInputFormat(Path filePath,
String lineDelimiter,
String fieldDelimiter,
PojoTypeInfo<OUT> pojoTypeInfo,
boolean[] includedFieldsMask) |
PojoCsvInputFormat(Path filePath,
String lineDelimiter,
String fieldDelimiter,
PojoTypeInfo<OUT> pojoTypeInfo,
int[] includedFieldsMask) |
PojoCsvInputFormat(Path filePath,
String lineDelimiter,
String fieldDelimiter,
PojoTypeInfo<OUT> pojoTypeInfo,
String[] fieldNames) |
PojoCsvInputFormat(Path filePath,
String lineDelimiter,
String fieldDelimiter,
PojoTypeInfo<OUT> pojoTypeInfo,
String[] fieldNames,
boolean[] includedFieldsMask) |
PojoCsvInputFormat(Path filePath,
String lineDelimiter,
String fieldDelimiter,
PojoTypeInfo<OUT> pojoTypeInfo,
String[] fieldNames,
int[] includedFieldsMask) |
PrimitiveInputFormat(Path filePath,
Class<OT> primitiveClass) |
PrimitiveInputFormat(Path filePath,
String delimiter,
Class<OT> primitiveClass) |
RowCsvInputFormat(Path filePath,
TypeInformation[] fieldTypes) |
RowCsvInputFormat(Path filePath,
TypeInformation[] fieldTypes,
boolean emptyColumnAsNull) |
RowCsvInputFormat(Path filePath,
TypeInformation[] fieldTypes,
int[] selectedFields) |
RowCsvInputFormat(Path filePath,
TypeInformation[] fieldTypes,
String lineDelimiter,
String fieldDelimiter) |
RowCsvInputFormat(Path filePath,
TypeInformation[] fieldTypes,
String lineDelimiter,
String fieldDelimiter,
int[] selectedFields) |
RowCsvInputFormat(Path filePath,
TypeInformation[] fieldTypeInfos,
String lineDelimiter,
String fieldDelimiter,
int[] selectedFields,
boolean emptyColumnAsNull) |
TextInputFormat(Path filePath) |
TextOutputFormat(Path outputPath) |
TextOutputFormat(Path outputPath,
String charset) |
TextValueInputFormat(Path filePath) |
TupleCsvInputFormat(Path filePath,
String lineDelimiter,
String fieldDelimiter,
TupleTypeInfoBase<OUT> tupleTypeInfo) |
TupleCsvInputFormat(Path filePath,
String lineDelimiter,
String fieldDelimiter,
TupleTypeInfoBase<OUT> tupleTypeInfo,
boolean[] includedFieldsMask) |
TupleCsvInputFormat(Path filePath,
String lineDelimiter,
String fieldDelimiter,
TupleTypeInfoBase<OUT> tupleTypeInfo,
int[] includedFieldsMask) |
TupleCsvInputFormat(Path filePath,
TupleTypeInfoBase<OUT> tupleTypeInfo) |
TupleCsvInputFormat(Path filePath,
TupleTypeInfoBase<OUT> tupleTypeInfo,
boolean[] includedFieldsMask) |
TupleCsvInputFormat(Path filePath,
TupleTypeInfoBase<OUT> tupleTypeInfo,
int[] includedFieldsMask) |
Constructor and Description |
---|
ScalaCsvOutputFormat(Path outputPath)
Creates an instance of CsvOutputFormat.
|
ScalaCsvOutputFormat(Path outputPath,
String fieldDelimiter)
Creates an instance of CsvOutputFormat.
|
ScalaCsvOutputFormat(Path outputPath,
String recordDelimiter,
String fieldDelimiter)
Creates an instance of CsvOutputFormat.
|
Modifier and Type | Method and Description |
---|---|
Path |
LimitedConnectionsFileSystem.getHomeDirectory() |
Path |
SafetyNetWrapperFileSystem.getHomeDirectory() |
abstract Path |
FileSystem.getHomeDirectory()
Returns the path of the user's home directory in this file system.
|
Path |
Path.getParent()
Returns the parent of a path, i.e., everything that precedes the last separator
or
null if at root. |
Path |
FileInputSplit.getPath()
Returns the path of the file containing this split's data.
|
Path |
FileStatus.getPath()
Returns the corresponding Path to the FileStatus.
|
Path |
LimitedConnectionsFileSystem.getWorkingDirectory() |
Path |
SafetyNetWrapperFileSystem.getWorkingDirectory() |
abstract Path |
FileSystem.getWorkingDirectory()
Returns the path of the file system's current working directory.
|
Path |
Path.makeQualified(FileSystem fs)
Returns a qualified path object.
|
Path |
Path.suffix(String suffix)
Adds a suffix to the final name in the path.
|
Modifier and Type | Method and Description |
---|---|
FSDataOutputStream |
FileSystem.create(Path f,
boolean overwrite)
Deprecated.
Use
FileSystem.create(Path, WriteMode) instead. |
FSDataOutputStream |
LimitedConnectionsFileSystem.create(Path f,
boolean overwrite,
int bufferSize,
short replication,
long blockSize)
Deprecated.
|
FSDataOutputStream |
SafetyNetWrapperFileSystem.create(Path f,
boolean overwrite,
int bufferSize,
short replication,
long blockSize) |
FSDataOutputStream |
FileSystem.create(Path f,
boolean overwrite,
int bufferSize,
short replication,
long blockSize)
Deprecated.
Deprecated because not well supported across types of file systems.
Control the behavior of specific file systems via configurations instead.
|
FSDataOutputStream |
LimitedConnectionsFileSystem.create(Path f,
FileSystem.WriteMode overwriteMode) |
FSDataOutputStream |
SafetyNetWrapperFileSystem.create(Path f,
FileSystem.WriteMode overwrite) |
abstract FSDataOutputStream |
FileSystem.create(Path f,
FileSystem.WriteMode overwriteMode)
Opens an FSDataOutputStream to a new file at the given path.
|
boolean |
LimitedConnectionsFileSystem.delete(Path f,
boolean recursive) |
boolean |
SafetyNetWrapperFileSystem.delete(Path f,
boolean recursive) |
abstract boolean |
FileSystem.delete(Path f,
boolean recursive)
Delete a file.
|
boolean |
LimitedConnectionsFileSystem.exists(Path f) |
boolean |
SafetyNetWrapperFileSystem.exists(Path f) |
boolean |
FileSystem.exists(Path f)
Check if exists.
|
FileStatus |
LimitedConnectionsFileSystem.getFileStatus(Path f) |
FileStatus |
SafetyNetWrapperFileSystem.getFileStatus(Path f) |
abstract FileStatus |
FileSystem.getFileStatus(Path f)
Return a file status object that represents the path.
|
boolean |
SafetyNetWrapperFileSystem.initOutPathDistFS(Path outPath,
FileSystem.WriteMode writeMode,
boolean createDirectory) |
boolean |
FileSystem.initOutPathDistFS(Path outPath,
FileSystem.WriteMode writeMode,
boolean createDirectory)
Initializes output directories on distributed file systems according to the given write mode.
|
boolean |
SafetyNetWrapperFileSystem.initOutPathLocalFS(Path outPath,
FileSystem.WriteMode writeMode,
boolean createDirectory) |
boolean |
FileSystem.initOutPathLocalFS(Path outPath,
FileSystem.WriteMode writeMode,
boolean createDirectory)
Initializes output directories on local file systems according to the given write mode.
|
FileStatus[] |
LimitedConnectionsFileSystem.listStatus(Path f) |
FileStatus[] |
SafetyNetWrapperFileSystem.listStatus(Path f) |
abstract FileStatus[] |
FileSystem.listStatus(Path f)
List the statuses of the files/directories in the given path if the path is
a directory.
|
boolean |
LimitedConnectionsFileSystem.mkdirs(Path f) |
boolean |
SafetyNetWrapperFileSystem.mkdirs(Path f) |
abstract boolean |
FileSystem.mkdirs(Path f)
Make the given file and all non-existent parents into directories.
|
FSDataInputStream |
LimitedConnectionsFileSystem.open(Path f) |
FSDataInputStream |
SafetyNetWrapperFileSystem.open(Path f) |
abstract FSDataInputStream |
FileSystem.open(Path f)
Opens an FSDataInputStream at the indicated Path.
|
FSDataInputStream |
LimitedConnectionsFileSystem.open(Path f,
int bufferSize) |
FSDataInputStream |
SafetyNetWrapperFileSystem.open(Path f,
int bufferSize) |
abstract FSDataInputStream |
FileSystem.open(Path f,
int bufferSize)
Opens an FSDataInputStream at the indicated Path.
|
boolean |
LimitedConnectionsFileSystem.rename(Path src,
Path dst) |
boolean |
SafetyNetWrapperFileSystem.rename(Path src,
Path dst) |
abstract boolean |
FileSystem.rename(Path src,
Path dst)
Renames the file/directory src to dst.
|
Constructor and Description |
---|
FileInputSplit(int num,
Path file,
long start,
long length,
String[] hosts)
Constructs a split with host information.
|
Path(Path parent,
Path child)
Resolve a child path against a parent path.
|
Path(Path parent,
String child)
Resolve a child path against a parent path.
|
Path(String parent,
Path child)
Resolve a child path against a parent path.
|
Modifier and Type | Method and Description |
---|---|
Path |
LocalFileSystem.getHomeDirectory() |
Path |
LocalFileStatus.getPath() |
Path |
LocalFileSystem.getWorkingDirectory() |
Modifier and Type | Method and Description |
---|---|
FSDataOutputStream |
LocalFileSystem.create(Path filePath,
FileSystem.WriteMode overwrite) |
boolean |
LocalFileSystem.delete(Path f,
boolean recursive) |
FileStatus |
LocalFileSystem.getFileStatus(Path f) |
FileStatus[] |
LocalFileSystem.listStatus(Path f) |
boolean |
LocalFileSystem.mkdirs(Path f)
Recursively creates the directory specified by the provided path.
|
FSDataInputStream |
LocalFileSystem.open(Path f) |
FSDataInputStream |
LocalFileSystem.open(Path f,
int bufferSize) |
boolean |
LocalFileSystem.rename(Path src,
Path dst) |
Modifier and Type | Method and Description |
---|---|
Path |
FileCopyTask.getPath() |
Constructor and Description |
---|
FileCopyTask(Path path,
String relativePath) |
Constructor and Description |
---|
AvroInputFormat(Path filePath,
Class<E> type) |
AvroOutputFormat(Path filePath,
Class<E> type) |
Constructor and Description |
---|
GraphCsvReader(Path edgePath,
ExecutionEnvironment context) |
GraphCsvReader(Path edgePath,
MapFunction<K,VV> mapper,
ExecutionEnvironment context) |
GraphCsvReader(Path vertexPath,
Path edgePath,
ExecutionEnvironment context) |
Modifier and Type | Method and Description |
---|---|
URL |
MesosArtifactServer.addPath(Path path,
Path remoteFile)
Adds a path to the artifact server.
|
void |
MesosArtifactServer.removePath(Path remoteFile) |
scala.Option<URL> |
MesosArtifactServer.resolve(Path remoteFile) |
scala.Option<URL> |
MesosArtifactResolver.resolve(Path remoteFile) |
Constructor and Description |
---|
VirtualFileServerHandler(Path path) |
Modifier and Type | Method and Description |
---|---|
static List<PermanentBlobKey> |
BlobClient.uploadJarFiles(InetSocketAddress serverAddress,
Configuration clientConfig,
JobID jobId,
List<Path> jars)
Uploads the JAR files to the
PermanentBlobService of the BlobServer at the
given address with HA as configured. |
Modifier and Type | Field and Description |
---|---|
Path |
ContainerSpecification.Artifact.dest |
Path |
ContainerSpecification.Artifact.Builder.dest |
Path |
ContainerSpecification.Artifact.source |
Path |
ContainerSpecification.Artifact.Builder.source |
Modifier and Type | Method and Description |
---|---|
ContainerSpecification.Artifact.Builder |
ContainerSpecification.Artifact.Builder.setDest(Path dest) |
ContainerSpecification.Artifact.Builder |
ContainerSpecification.Artifact.Builder.setSource(Path source) |
Constructor and Description |
---|
Artifact(Path source,
Path dest,
boolean executable,
boolean cachable,
boolean extract) |
Constructor and Description |
---|
KeytabOverlay(Path keytab) |
Krb5ConfOverlay(Path krb5Conf) |
Modifier and Type | Method and Description |
---|---|
Map<String,Future<Path>> |
Environment.getDistributedCacheEntries() |
Modifier and Type | Method and Description |
---|---|
Future<Path> |
FileCache.createTmpFile(String name,
DistributedCache.DistributedCacheEntry entry,
JobID jobID)
If the file doesn't exists locally, it will copy the file to the temp directory.
|
Modifier and Type | Method and Description |
---|---|
static void |
FileCache.copy(Path sourcePath,
Path targetPath,
boolean executable) |
Modifier and Type | Method and Description |
---|---|
Path |
HadoopFileSystem.getHomeDirectory() |
Path |
HadoopFileStatus.getPath() |
Path |
HadoopFileSystem.getWorkingDirectory() |
Modifier and Type | Method and Description |
---|---|
HadoopDataOutputStream |
HadoopFileSystem.create(Path f,
boolean overwrite,
int bufferSize,
short replication,
long blockSize) |
HadoopDataOutputStream |
HadoopFileSystem.create(Path f,
FileSystem.WriteMode overwrite) |
boolean |
HadoopFileSystem.delete(Path f,
boolean recursive) |
FileStatus |
HadoopFileSystem.getFileStatus(Path f) |
FileStatus[] |
HadoopFileSystem.listStatus(Path f) |
boolean |
HadoopFileSystem.mkdirs(Path f) |
HadoopDataInputStream |
HadoopFileSystem.open(Path f) |
HadoopDataInputStream |
HadoopFileSystem.open(Path f,
int bufferSize) |
boolean |
HadoopFileSystem.rename(Path src,
Path dst) |
Constructor and Description |
---|
FsNegativeRunningJobsRegistry(FileSystem fileSystem,
Path workingDirectory)
Creates a new registry that writes its files to the given FileSystem at
the given working directory path.
|
FsNegativeRunningJobsRegistry(Path workingDirectory)
Creates a new registry that writes to the FileSystem and working directory
denoted by the given path.
|
Modifier and Type | Method and Description |
---|---|
static Path |
FsJobArchivist.archiveJob(Path rootPath,
AccessExecutionGraph graph)
Writes the given
AccessExecutionGraph to the FileSystem pointed to by
JobManagerOptions.ARCHIVE_DIR . |
Modifier and Type | Method and Description |
---|---|
static Path |
FsJobArchivist.archiveJob(Path rootPath,
AccessExecutionGraph graph)
Writes the given
AccessExecutionGraph to the FileSystem pointed to by
JobManagerOptions.ARCHIVE_DIR . |
static Collection<ArchivedJson> |
FsJobArchivist.getArchivedJsons(Path file)
Reads the given archive file and returns a
Collection of contained ArchivedJson . |
Modifier and Type | Method and Description |
---|---|
List<Path> |
JobGraph.getUserJars()
Gets the list of assigned user jar paths.
|
Modifier and Type | Method and Description |
---|---|
void |
JobGraph.addJar(Path jar)
Adds the path of a JAR file required to run the job on a task manager.
|
Constructor and Description |
---|
DistributedRuntimeUDFContext(TaskInfo taskInfo,
ClassLoader userCodeClassLoader,
ExecutionConfig executionConfig,
Map<String,Future<Path>> cpTasks,
Map<String,Accumulator<?,?>> accumulators,
MetricGroup metrics) |
Constructor and Description |
---|
RetrievableStreamStateHandle(Path filePath,
long stateSize) |
Modifier and Type | Method and Description |
---|---|
protected Path |
FsSavepointStreamFactory.createBasePath(FileSystem fs,
Path checkpointDirectory,
JobID jobID) |
protected Path |
FsCheckpointStreamFactory.createBasePath(FileSystem fs,
Path checkpointDirectory,
JobID jobID) |
protected Path |
FsSavepointStreamFactory.createCheckpointDirPath(Path checkpointDirectory,
long checkpointID) |
protected Path |
FsCheckpointStreamFactory.createCheckpointDirPath(Path checkpointDirectory,
long checkpointID) |
Path |
FsStateBackend.getBasePath()
Gets the base directory where all state-containing files are stored.
|
Path |
FileStateHandle.getFilePath()
Gets the path where this handle's state is stored.
|
Modifier and Type | Method and Description |
---|---|
protected Path |
FsSavepointStreamFactory.createBasePath(FileSystem fs,
Path checkpointDirectory,
JobID jobID) |
protected Path |
FsCheckpointStreamFactory.createBasePath(FileSystem fs,
Path checkpointDirectory,
JobID jobID) |
protected Path |
FsSavepointStreamFactory.createCheckpointDirPath(Path checkpointDirectory,
long checkpointID) |
protected Path |
FsCheckpointStreamFactory.createCheckpointDirPath(Path checkpointDirectory,
long checkpointID) |
Constructor and Description |
---|
FileStateHandle(Path filePath,
long stateSize)
Creates a new file state for the given file path.
|
FsCheckpointStateOutputStream(Path basePath,
FileSystem fs,
int bufferSize,
int localStateThreshold) |
FsCheckpointStreamFactory(Path checkpointDataUri,
JobID jobId,
int fileStateSizeThreshold)
Creates a new state backend that stores its checkpoint data in the file system and location
defined by the given URI.
|
FsSavepointStreamFactory(Path checkpointDataUri,
JobID jobId,
int fileStateSizeThreshold) |
FsStateBackend(Path checkpointDataUri)
Creates a new state backend that stores its checkpoint data in the file system and location
defined by the given URI.
|
FsStateBackend(Path checkpointDataUri,
boolean asynchronousSnapshots)
Creates a new state backend that stores its checkpoint data in the file system and location
defined by the given URI.
|
Modifier and Type | Method and Description |
---|---|
Map<String,Future<Path>> |
RuntimeEnvironment.getDistributedCacheEntries() |
Constructor and Description |
---|
RuntimeEnvironment(JobID jobId,
JobVertexID jobVertexId,
ExecutionAttemptID executionId,
ExecutionConfig executionConfig,
TaskInfo taskInfo,
Configuration jobConfiguration,
Configuration taskConfiguration,
ClassLoader userCodeClassLoader,
MemoryManager memManager,
IOManager ioManager,
BroadcastVariableManager bcVarManager,
AccumulatorRegistry accumulatorRegistry,
TaskKvStateRegistry kvStateRegistry,
InputSplitProvider splitProvider,
Map<String,Future<Path>> distCacheEntries,
ResultPartitionWriter[] writers,
InputGate[] inputGates,
CheckpointResponder checkpointResponder,
TaskManagerRuntimeInfo taskManagerInfo,
TaskMetricGroup metrics,
Task containingTask) |
Modifier and Type | Method and Description |
---|---|
static Path |
WebMonitorUtils.validateAndNormalizeUri(URI archiveDirUri)
Checks and normalizes the given URI.
|
Constructor and Description |
---|
FileSystemStateStorageHelper(Path rootPath,
String prefix) |
Constructor and Description |
---|
TimestampedFileInputSplit(long modificationTime,
int num,
Path file,
long start,
long length,
String[] hosts)
Creates a
TimestampedFileInputSplit based on the file modification time and
the rest of the information of the FileInputSplit , as returned by the
underlying filesystem. |
Modifier and Type | Method and Description |
---|---|
static void |
TestStreamEnvironment.setAsContext(org.apache.flink.runtime.minicluster.LocalFlinkMiniCluster cluster,
int parallelism,
Collection<Path> jarFiles,
Collection<URL> classpaths)
Sets the streaming context environment to a TestStreamEnvironment that runs its programs on
the given cluster with the given default parallelism and the specified jar files and class
paths.
|
Constructor and Description |
---|
TestStreamEnvironment(org.apache.flink.runtime.minicluster.LocalFlinkMiniCluster miniCluster,
int parallelism,
Collection<Path> jarFiles,
Collection<URL> classPaths) |
Modifier and Type | Method and Description |
---|---|
static void |
TestEnvironment.setAsContext(org.apache.flink.runtime.minicluster.LocalFlinkMiniCluster miniCluster,
int parallelism,
Collection<Path> jarFiles,
Collection<URL> classPaths)
Sets the current
ExecutionEnvironment to be a TestEnvironment . |
Constructor and Description |
---|
TestEnvironment(org.apache.flink.runtime.minicluster.LocalFlinkMiniCluster miniCluster,
int parallelism,
boolean isObjectReuseEnabled,
Collection<Path> jarFiles,
Collection<URL> classPaths) |
Modifier and Type | Method and Description |
---|---|
static boolean |
FileUtils.deletePathIfEmpty(FileSystem fileSystem,
Path path)
Deletes the path if it is empty.
|
Modifier and Type | Field and Description |
---|---|
protected Path |
YarnHighAvailabilityServices.haDataDirectory
The directory for HA persistent data.
|
protected Path |
YarnHighAvailabilityServices.workingDirectory
The working directory of this YARN application.
|
Copyright © 2014–2018 The Apache Software Foundation. All rights reserved.