public class LinuxContainerExecutor extends ContainerExecutor
This class provides Container execution using a native
container-executor binary. By using a helper written it native code,
this class is able to do several things that the
DefaultContainerExecutor cannot, such as execution of applications
as the applications' owners, provide localization that takes advantage of
mapping the application owner to a UID on the execution host, resource
management through Linux CGROUPS, and Docker support.
If hadoop.security.authentication is set to simple,
then the
yarn.nodemanager.linux-container-executor.nonsecure-mode.limit-users
property will determine whether the LinuxContainerExecutor runs
processes as the application owner or as the default user, as set in the
yarn.nodemanager.linux-container-executor.nonsecure-mode.local-user
property.
The LinuxContainerExecutor will manage applications through an
appropriate LinuxContainerRuntime instance. This class uses a
DelegatingLinuxContainerRuntime instance, which will delegate calls
to either a DefaultLinuxContainerRuntime instance or a
DockerLinuxContainerRuntime instance, depending on the job's
configuration.
| Modifier and Type | Class and Description |
|---|---|
static class |
LinuxContainerExecutor.ExitCode
The container exit code.
|
ContainerExecutor.DelayedProcessKiller, ContainerExecutor.SignalDIRECTORY_CONTENTS, TASK_LAUNCH_SCRIPT_PERMISSION, TOKEN_FILE_NAME_FMT, WILDCARD| Constructor and Description |
|---|
LinuxContainerExecutor()
Default constructor to allow for creation through reflection.
|
LinuxContainerExecutor(LinuxContainerRuntime linuxContainerRuntime)
Create a LinuxContainerExecutor with a provided
LinuxContainerRuntime. |
| Modifier and Type | Method and Description |
|---|---|
protected void |
addSchedPriorityCommand(List<String> command)
Add a niceness level to the process that will be executed.
|
void |
buildMainArgs(List<String> command,
String user,
String appId,
String locId,
InetSocketAddress nmAddr,
String tokenFileName,
List<String> localDirs)
Set up the
ContainerLocalizer. |
void |
deleteAsUser(DeletionAsUserContext ctx)
Delete specified directories as a given user.
|
protected String |
getContainerExecutorExecutablePath(org.apache.hadoop.conf.Configuration conf)
Get the path to the
container-executor binary. |
String[] |
getIpAndHost(Container container) |
protected PrivilegedOperationExecutor |
getPrivilegedOperationExecutor() |
ResourceHandler |
getResourceHandler() |
void |
init(Context nmContext)
Run the executor initialization steps.
|
boolean |
isContainerAlive(ContainerLivenessContext ctx)
Check if a container is alive.
|
int |
launchContainer(ContainerStartContext ctx)
Launch the container on the node.
|
void |
mountCgroups(List<String> cgroupKVs,
String hierarchy)
Mount a CGROUPS controller at the requested mount point and create
a hierarchy for the NodeManager to manage.
|
int |
reacquireContainer(ContainerReacquisitionContext ctx)
Recover an already existing container.
|
protected File[] |
readDirAsUser(String user,
org.apache.hadoop.fs.Path dir)
Return the files in the target directory.
|
void |
setConf(org.apache.hadoop.conf.Configuration conf) |
boolean |
signalContainer(ContainerSignalContext ctx)
Signal container with the specified signal.
|
void |
startLocalizer(LocalizerStartContext ctx)
Prepare the environment for containers in this application to execute.
|
void |
symLink(String target,
String symlink)
Create a symlink file which points to the target.
|
activateContainer, deactivateContainer, getConf, getLocalIpAndHost, getNMEnvVar, getPidFilePath, getProcessId, getRunCommand, getRunCommand, getRunCommandForOther, getRunCommandForWindows, isContainerActive, localizeClasspathJar, logOutput, pauseContainer, resumeContainer, writeLaunchEnv, writeLaunchEnvpublic LinuxContainerExecutor()
public LinuxContainerExecutor(LinuxContainerRuntime linuxContainerRuntime)
LinuxContainerRuntime. Used primarily for testing.linuxContainerRuntime - the runtime to usepublic void setConf(org.apache.hadoop.conf.Configuration conf)
setConf in interface org.apache.hadoop.conf.ConfigurablesetConf in class ContainerExecutorprotected String getContainerExecutorExecutablePath(org.apache.hadoop.conf.Configuration conf)
container-executor binary. The path will
be absolute.conf - the Configurationcontainer-executor binaryprotected void addSchedPriorityCommand(List<String> command)
-n <nice> to the given command. The niceness level will be
taken from the
YarnConfiguration.NM_CONTAINER_EXECUTOR_SCHED_PRIORITY property.command - the command to which to add the niceness setting.protected PrivilegedOperationExecutor getPrivilegedOperationExecutor()
public void init(Context nmContext) throws IOException
ContainerExecutorinit in class ContainerExecutornmContext - Context of NMIOException - if initialization failspublic void startLocalizer(LocalizerStartContext ctx) throws IOException, InterruptedException
ContainerExecutorFor $x in local.dirs create $x/$user/$appId Copy $nmLocal/appTokens -> $N/$user/$appId For $rsrc in private resources Copy $rsrc -> $N/$user/filecache/[idef] For $rsrc in job resources Copy $rsrc -> $N/$user/$appId/filecache/idef
startLocalizer in class ContainerExecutorctx - LocalizerStartContext that encapsulates necessary information
for starting a localizer.IOException - for most application init failuresInterruptedException - if application init thread is halted by NMpublic void buildMainArgs(List<String> command, String user, String appId, String locId, InetSocketAddress nmAddr, String tokenFileName, List<String> localDirs)
ContainerLocalizer.command - the current ShellCommandExecutor command lineuser - localization userappId - localized app idlocId - localizer idnmAddr - nodemanager addresslocalDirs - list of local dirsContainerLocalizer.buildMainArgs(java.util.List<java.lang.String>, java.lang.String, java.lang.String, java.lang.String, java.net.InetSocketAddress, java.lang.String, java.util.List<java.lang.String>, org.apache.hadoop.conf.Configuration)public int launchContainer(ContainerStartContext ctx) throws IOException, org.apache.hadoop.yarn.exceptions.ConfigurationException
ContainerExecutorlaunchContainer in class ContainerExecutorctx - Encapsulates information necessary for launching containers.IOException - if the container launch failsorg.apache.hadoop.yarn.exceptions.ConfigurationException - if config error was foundpublic String[] getIpAndHost(Container container) throws ContainerExecutionException
getIpAndHost in class ContainerExecutorContainerExecutionExceptionpublic int reacquireContainer(ContainerReacquisitionContext ctx) throws IOException, InterruptedException
ContainerExecutorreacquireContainer in class ContainerExecutorctx - encapsulates information necessary to reacquire containerIOException - if there is a failure while reacquiring the containerInterruptedException - if interrupted while waiting to reacquire
the containerpublic boolean signalContainer(ContainerSignalContext ctx) throws IOException
ContainerExecutorsignalContainer in class ContainerExecutorctx - Encapsulates information necessary for signaling containers.IOException - if signaling the container failspublic void deleteAsUser(DeletionAsUserContext ctx)
ContainerExecutordeleteAsUser in class ContainerExecutorctx - Encapsulates information necessary for deletion.protected File[] readDirAsUser(String user, org.apache.hadoop.fs.Path dir)
ContainerExecutorreadDirAsUser in class ContainerExecutoruser - the user as whom to access the target directorydir - the target directorypublic void symLink(String target, String symlink)
ContainerExecutorsymLink in class ContainerExecutortarget - The target for symlinksymlink - the symlink filepublic boolean isContainerAlive(ContainerLivenessContext ctx) throws IOException
ContainerExecutorisContainerAlive in class ContainerExecutorctx - Encapsulates information necessary for container liveness check.IOException - if there is a failure while checking the container
statuspublic void mountCgroups(List<String> cgroupKVs, String hierarchy) throws IOException
cgroupKVs - a key-value pair of the form
controller=mount-pathhierarchy - the top directory of the hierarchy for the NodeManagerIOException - if there is a problem mounting the CGROUPSpublic ResourceHandler getResourceHandler()
Copyright © 2022 Apache Software Foundation. All rights reserved.