YARN

1. What is YARN

Yet Another Resource Negotiator (another resource coordinator) is a new Hadoop resource manager, which is a general resource management system that provides unified resource management and scheduling for upper-layer applications .

2. YARN Architecture

1. ResurceManager(RM) : A pure scheduler dedicated to the allocation and management of resources available in the cluster.
2. Container : The abstract representation of resources allocated to specific applications, including memory, cpu, disk
3. NodeManager(NM) : Responsible for the management of local resources of nodes, including starting Containers of applications, monitoring their resource usage, and reporting to RM
4. App Master (ApplicationMaster(AM)) : An instance of a specific framework library, responsible for negotiating resources with RM, and coordinating with NM to execute and monitor Containers and their resource consumption. AM also runs as a Container.
5. Client (Client) : is an instance in the cluster that can submit applications to RM, and specifies the AM type required to execute the application

3. How to write a YARN application

1. Client

   • Initialize and start a YarnClient

   Configuration yarnConfig = new YarnConfiguration(getConf());
   YarnClient client = YarnClient.createYarnClient();
   client.init(yarnConfig);
   client.start();
   

   • Create an application

   YarnClientApplication app = client.createApplication();
   GetNewApplicationResponse appResponse = app.getNewApplicationResponse();
   

   • Set the application submission context

   // 1. 设置应用程序提交上下文基本信息
   ApplicationSubmissionContext appContext = app.getApplicationSubmissionContext();
   appContext.setApplicationId(appResponse.getApplicationId());
   appContext.setApplicationName(config.getProperty("app.name"));
   appContext.setApplicationType(config.getProperty("app.type"));
   appContext.setApplicationTags(new LinkedHashSet<>(Arrays.asList(config.getProperty("app.tags").split(","))));
   // queue：默认是default
   appContext.setQueue(config.getProperty("app.queue"));
   appContext.setPriority(Priority.newInstance(Integer.parseInt(config.getProperty("app.priority"))));
   appContext.setResource(Resource.newInstance(Integer.parseInt(config.getProperty("am.memory")),
       Integer.parseInt(config.getProperty("am.vCores"))));
   
   //2. 设置am container启动上下文 
   ContainerLaunchContext amContainer = Records.newRecord(ContainerLaunchContext.class);
   
   // 3. 设置am localResources
   Map<String, LocalResource> amLocalResources = new LinkedHashMap<>();
   LocalResource drillArchive = Records.newRecord(LocalResource.class);
   drillArchive.setResource(ConverterUtils.getYarnUrlFromPath(drillArchiveFileStatus.getPath()));
   drillArchive.setSize(drillArchiveFileStatus.getLen());
   drillArchive.setTimestamp(drillArchiveFileStatus.getModificationTime());
   drillArchive.setType(LocalResourceType.ARCHIVE);
   drillArchive.setVisibility(LocalResourceVisibility.PUBLIC);
   amLocalResources.put(config.getProperty("drill.archive.name"), drillArchive);
   amContainer.setLocalResources(amLocalResources);
   
   // 4. 设置am environment
    Map<String, String> amEnvironment = new LinkedHashMap<>();
   // add Hadoop Classpath
   for (String classpath : yarnConfig.getStrings(YarnConfiguration.YARN_APPLICATION_CLASSPATH,
       YarnConfiguration.DEFAULT_YARN_APPLICATION_CLASSPATH)) {
       Apps.addToEnvironment(amEnvironment, Environment.CLASSPATH.name(),
           classpath.trim(), ApplicationConstants.CLASS_PATH_SEPARATOR);
   }
   Apps.addToEnvironment(amEnvironment, Environment.CLASSPATH.name(),
       Environment.PWD.$() + File.separator + "*", ApplicationConstants.CLASS_PATH_SEPARATOR);
   StringWriter sw = new StringWriter();
   config.store(sw, "");
   String configBase64Binary = DatatypeConverter.printBase64Binary(sw.toString().getBytes("UTF-8"));
   Apps.addToEnvironment(amEnvironment, "DRILL_ON_YARN_CONFIG", configBase64Binary,
       ApplicationConstants.CLASS_PATH_SEPARATOR);
   amContainer.setEnvironment(amEnvironment);
   
   // 5. 设置am command
   
   List<String> commands = new ArrayList<>();
   commands.add(Environment.SHELL.$$());
   commands.add(config.getProperty("drill.archive.name") + "/bin/drill-am.sh");
   commands.add("1>" + ApplicationConstants.LOG_DIR_EXPANSION_VAR + "/" + ApplicationConstants.STDOUT);
   commands.add("2>" + ApplicationConstants.LOG_DIR_EXPANSION_VAR + "/" + ApplicationConstants.STDERR);
   StringBuilder amCommand = new StringBuilder();
   for (String str : commands) {
       amCommand.append(str).append(" ");
   }
   amCommand.setLength(amCommand.length() - " ".length());
   amContainer.setCommands(Collections.singletonList(amCommand.toString()));
   
   // 6. 设置安全令牌
   if (UserGroupInformation.isSecurityEnabled()) {
       Credentials credentials = new Credentials();
       String tokenRenewer = yarnConfig.get(YarnConfiguration.RM_PRINCIPAL);
       final Token<?> tokens[] = fileSystem.addDelegationTokens(tokenRenewer, credentials);
       DataOutputBuffer dob = new DataOutputBuffer();
       credentials.writeTokenStorageToStream(dob);
       ByteBuffer fsTokens = ByteBuffer.wrap(dob.getData(), 0, dob.getLength());
       amContainer.setTokens(fsTokens);
   }
   
   appContext.setAMContainerSpec(amContainer);
   

   • Submit application

   client.submitApplication(appContext);
   

2. ApplicationMaster(AM)

   1. Initialize AMRMClientAsync

   YarnConfiguration yarnConfig = new YarnConfiguration();
   AMRMClientAsync amrmClientAsync = AMRMClientAsync.createAMRMClientAsync(5000, new AMRMCallbackHandler());
   amrmClientAsync.init(yarnConfig);
   amrmClientAsync.start();
   

   2. Initialize NMClientAsync

   YarnConfiguration yarnConfig = new YarnConfiguration();
   NMClientAsync nmClientAsync = NMClientAsync.createNMClientAsync(new NMCallbackHandler());
   nmClientAsync.init(yarnConfig);
   nmClientAsync.start();
   

   3. Register ApplicationMaster(AM)

   String thisHostName = InetAddress.getLocalHost();
   amrmClientAsync.registerApplicationMaster(thisHostName, 0, "");
   

   4. add ContainerRequest

   for (NodeReport containerReport : containerReports) {
       ContainerRequest containerRequest = new ContainerRequest(capability,
           new String[] {containerReport.getNodeId().getHost()},
           null, priority, false);
       amrmClientAsync.addContainerRequest(containerRequest);
   }
   

   5. start the container

   private static class AMRMCallbackHandler implements AMRMClientAsync.CallbackHandler {
       @Override
       public void onContainersAllocated(List<Container> containers) {
           for (Container container : containers) {
               ContainerLaunchContext containerContext = Records.newRecord(ContainerLaunchContext.class);
   
               // setEnvironment
               Map<String, String> containerEnvironment = new LinkedHashMap<>();
               // add Hadoop Classpath
               for (String classpath : yarnConfig.getStrings(YarnConfiguration.YARN_APPLICATION_CLASSPATH,
                   YarnConfiguration.DEFAULT_YARN_APPLICATION_CLASSPATH)) {
                   Apps.addToEnvironment(containerEnvironment, Environment.CLASSPATH.name(),
                       classpath.trim(), ApplicationConstants.CLASS_PATH_SEPARATOR);
               }
               Apps.addToEnvironment(containerEnvironment, Environment.CLASSPATH.name(),
                   Environment.PWD.$() + File.separator + "*", ApplicationConstants.CLASS_PATH_SEPARATOR);
               containerContext.setEnvironment(containerEnvironment);
   
               // setContainerResource
               Map<String, LocalResource> containerResources = new LinkedHashMap<>();
               LocalResource drillArchive = Records.newRecord(LocalResource.class);
               String drillArchivePath = appConfig.getProperty("fs.upload.dir") + appConfig.getProperty(
                   "drill.archive.name");
               Path path = new Path(drillArchivePath);
               FileStatus fileStatus = FileSystem.get(yarnConfig).getFileStatus(path);
               drillArchive.setResource(ConverterUtils.getYarnUrlFromPath(fileStatus.getPath()));
               drillArchive.setSize(fileStatus.getLen());
               drillArchive.setTimestamp(fileStatus.getModificationTime());
               drillArchive.setType(LocalResourceType.ARCHIVE);
               drillArchive.setVisibility(LocalResourceVisibility.PUBLIC);
               containerResources.put(appConfig.getProperty("drill.archive.name"), drillArchive);
               containerContext.setLocalResources(containerResources);
   
               // setContainerCommand
               List<String> commands = new ArrayList<>();
               commands.add(Environment.SHELL.$$());
               commands.add(appConfig.getProperty("drill.archive.name") + "/bin/drillbit.sh run");
               commands.add("1>" + ApplicationConstants.LOG_DIR_EXPANSION_VAR + "/" + ApplicationConstants.STDOUT);
               commands.add("2>" + ApplicationConstants.LOG_DIR_EXPANSION_VAR + "/" + ApplicationConstants.STDERR);
               StringBuilder containerCommand = new StringBuilder();
               for (String str : commands) {
                   containerCommand.append(str).append(" ");
               }
               containerCommand.setLength(containerCommand.length() - " ".length());
               containerContext.setCommands(Collections.singletonList(containerCommand.toString()));
   
               nmClientAsync.startContainerAsync(container, containerContext);
           }
   
       }
   }
   

   6. unregisterApplicationMaster(AM)

   amrmClientAsync.unregisterApplicationMaster(appStatus, appMessage, null);
   

5. Why YARN

1. Scalability: Practice has proved that it is extremely difficult to expand JobTracker to 4000 nodes in MRv1, because JobTarcker undertakes too many responsibilities, including resource scheduling, task tracking and monitoring. JobTracker programming is getting overwhelmed. In YARN, JobTarcker is split into RM and AM, with clearer responsibilities, less responsibilities for each component, and more lightweight.
2. Support for programming model diversity: Due to the design of JobTarcker and TaskTarckerde in MRv1 and the coupled verification of the MR framework, MRv1 only supports the mapreduce computing framework, but does not support parallel computing, stream computing, and memory computing. In YARN, the AM is responsible for the specific framework, which is controlled by the application itself. YARN provides unified resource scheduling.
3. Framework upgrade is easier: In MRv1, if you want to upgrade the MR framework, you need to restart the entire Hadoop cluster, which is very risky. In YARN, AM is responsible for the specific framework, which is controlled by the application itself, so it is only necessary to upgrade the user program library.
4. Cluster resource utilization: MRv1 introduces the concept of "slot" to represent the computing resources on each node, and divides the resources (CPU, memory, disk, etc.) on each node into several equal parts, each part is represented by a slot, and at the same time It is stipulated that a task can occupy multiple slots according to actual needs. Slots are divided into two types: Mapslot and Reduceslot, but sharing is not allowed. For a job, at the beginning of running, the Map slot resources are scarce and the Reduce slots are idle. When all the Map Tasks are run, the Reduce slots are scarce and the Map slots are idle, which obviously reduces the resource utilization. In YARN, resources are represented in the form of Containers, including memory, CPU, etc. Compared with MRv1, the resource abstraction granularity is finer. Secondly, the flexibility of resources is guaranteed through RM's Scheduler (FIFO, Capacity, Fair).

6. References

1. HadoopYARN Permissions Guide
2. "Inside Hadoop Technology In-depth Analysis of YARN Architecture Design and Implementation Principles"