public final class Utils extends Object
Modifier and Type | Field and Description |
---|---|
static String |
KRB5_FILE_NAME
KRB5 file name populated in YARN container for secure IT run.
|
static String |
YARN_SITE_FILE_NAME
Yarn site xml file name populated in YARN container for secure IT run.
|
Modifier and Type | Method and Description |
---|---|
static void |
addToEnvironment(Map<String,String> environment,
String variable,
String value)
Copied method from org.apache.hadoop.yarn.util.Apps.
|
static void |
concatWithSpace(StringBuilder sb,
String value) |
static void |
deleteApplicationFiles(String applicationFilesDir)
Deletes the YARN application files, e.g., Flink binaries, libraries, etc., from the remote
filesystem.
|
static String |
generateJvmOptsString(Configuration conf,
List<ConfigOption<String>> jvmOptions,
boolean hasKrb5) |
static org.apache.hadoop.fs.Path |
getPathFromLocalFile(File localFile) |
static org.apache.hadoop.fs.Path |
getPathFromLocalFilePathStr(String localPathStr) |
static List<org.apache.hadoop.fs.Path> |
getQualifiedRemoteProvidedLibDirs(Configuration configuration,
org.apache.hadoop.yarn.conf.YarnConfiguration yarnConfiguration) |
static Optional<org.apache.hadoop.fs.Path> |
getQualifiedRemoteProvidedUsrLib(Configuration configuration,
org.apache.hadoop.yarn.conf.YarnConfiguration yarnConfiguration) |
static String |
getStartCommand(String template,
Map<String,String> startCommandValues)
Replaces placeholders in the template start command with values from startCommandValues.
|
static String |
getTaskManagerShellCommand(Configuration flinkConfig,
ContaineredTaskManagerParameters tmParams,
String configDirectory,
String logDirectory,
boolean hasLogback,
boolean hasLog4j,
boolean hasKrb5,
Class<?> mainClass,
String mainArgs)
Generates the shell command to start a task manager.
|
static org.apache.hadoop.yarn.conf.YarnConfiguration |
getYarnAndHadoopConfiguration(Configuration flinkConfig) |
static org.apache.hadoop.yarn.conf.YarnConfiguration |
getYarnConfiguration(Configuration flinkConfig)
Add additional config entries from the flink config to the yarn config.
|
static boolean |
isUsrLibDirectory(org.apache.hadoop.fs.FileSystem fileSystem,
org.apache.hadoop.fs.Path path) |
static String |
resolveKeytabPath(String workingDir,
String keytabPath)
Resolve keytab path either as absolute path or relative to working directory.
|
static void |
setAclsFor(org.apache.hadoop.yarn.api.records.ContainerLaunchContext amContainer,
Configuration flinkConfig)
Sets the application ACLs for the given ContainerLaunchContext based on the values specified
in the given Flink configuration.
|
static void |
setupYarnClassPath(org.apache.hadoop.conf.Configuration conf,
Map<String,String> appMasterEnv) |
public static final String KRB5_FILE_NAME
public static final String YARN_SITE_FILE_NAME
public static void setupYarnClassPath(org.apache.hadoop.conf.Configuration conf, Map<String,String> appMasterEnv)
public static void deleteApplicationFiles(String applicationFilesDir)
applicationFilesDir
- The application files directory.public static void addToEnvironment(Map<String,String> environment, String variable, String value)
public static String resolveKeytabPath(String workingDir, String keytabPath)
workingDir
- current working directorykeytabPath
- configured keytab path.public static String getTaskManagerShellCommand(Configuration flinkConfig, ContaineredTaskManagerParameters tmParams, String configDirectory, String logDirectory, boolean hasLogback, boolean hasLog4j, boolean hasKrb5, Class<?> mainClass, String mainArgs)
flinkConfig
- The Flink configuration.tmParams
- Parameters for the task manager.configDirectory
- The configuration directory for the config.yamllogDirectory
- The log directory.hasLogback
- Uses logback?hasLog4j
- Uses log4j?mainClass
- The main class to start with.public static String getStartCommand(String template, Map<String,String> startCommandValues)
If the default template ConfigConstants.DEFAULT_YARN_CONTAINER_START_COMMAND_TEMPLATE
is used, the following keys
must be present in the map or the resulting command will still contain placeholders:
template
- a template start command with placeholdersstartCommandValues
- a replacement map placeholder -> valuepublic static String generateJvmOptsString(Configuration conf, List<ConfigOption<String>> jvmOptions, boolean hasKrb5)
public static List<org.apache.hadoop.fs.Path> getQualifiedRemoteProvidedLibDirs(Configuration configuration, org.apache.hadoop.yarn.conf.YarnConfiguration yarnConfiguration) throws IOException
IOException
public static boolean isUsrLibDirectory(org.apache.hadoop.fs.FileSystem fileSystem, org.apache.hadoop.fs.Path path) throws IOException
IOException
public static Optional<org.apache.hadoop.fs.Path> getQualifiedRemoteProvidedUsrLib(Configuration configuration, org.apache.hadoop.yarn.conf.YarnConfiguration yarnConfiguration) throws IOException, IllegalArgumentException
IOException
IllegalArgumentException
public static org.apache.hadoop.yarn.conf.YarnConfiguration getYarnAndHadoopConfiguration(Configuration flinkConfig)
public static org.apache.hadoop.yarn.conf.YarnConfiguration getYarnConfiguration(Configuration flinkConfig)
flinkConfig
- The Flink configuration object.public static void setAclsFor(org.apache.hadoop.yarn.api.records.ContainerLaunchContext amContainer, Configuration flinkConfig)
amContainer
- the ContainerLaunchContext to set the ACLs for.flinkConfig
- the Flink configuration to read the ACL values from.public static org.apache.hadoop.fs.Path getPathFromLocalFile(File localFile)
public static org.apache.hadoop.fs.Path getPathFromLocalFilePathStr(String localPathStr)
public static void concatWithSpace(StringBuilder sb, String value)
Copyright © 2014–2024 The Apache Software Foundation. All rights reserved.