The original map-reduce architecture is simple and clear. In the first few years of its launch, it has also received numerous successful cases and won wide support and affirmation from the industry. However, with the growth of the scale of distributed system clusters and their workloads, The problems of the original framework gradually surfaced, and the main problems are as follows:
1) The JobTracker is the centralized processing point of Map-reduce and has a single point of failure.
2) JobTracker has completed too many tasks, resulting in excessive resource consumption. When there are too many map-reduce jobs, it will cause a lot of memory overhead, and potentially increase the risk of JobTracker fail, which is also The industry generally concludes that the old Hadoop Map-Reduce can only support the upper limit of 4000 node hosts.
3) On the TaskTracker side, it is too simple to use the number of map/reduce tasks as the resource representation, and does not take into account the occupancy of cpu/memory. If two tasks with large memory consumption are scheduled together, OOM is prone to occur.
4) On the TaskTracker side, the resources are forcibly divided into map task slots and reduce task slots. If there are only map tasks or only reduce tasks in the system, it will cause waste of resources, which is the problem of cluster resource utilization mentioned above. .
5) When analyzing the source code, it will be found that the code is very difficult to read, often because a class has done too many things, and the amount of code is more than 3000 lines, which makes the task of the class unclear and increases bug fixes and version maintenance. difficulty.
6) From an operational point of view, the current Hadoop MapReduce framework forces a system-level upgrade whenever there are any important or unimportant changes (such as bug fixes, performance improvements, and characterizations). Worse yet, it forces every client of the distributed cluster system to update at the same time, regardless of user preferences. These updates can cause users to waste a lot of time verifying that their previous applications work with the new Hadoop version.
From the changing trend of using distributed systems in the industry and the long-term development of hadoop framework, MapReduce's JobTracker/TaskTracker mechanism needs large-scale adjustments to fix its shortcomings in scalability, memory consumption, threading model, reliability and performance . Over the past few years, the hadoop development team has made some bug fixes, but these fixes have become more expensive recently, which shows how difficult it is to make changes to the original framework.
In order to fundamentally solve the performance bottleneck of the old MapReduce framework and promote the long-term development of the Hadoop framework, starting from version 0.23.0, the MapReduce framework of Hadoop has been completely refactored and has undergone fundamental changes. New Hadoop MapReduce framework named MapReduceV2 or Yarn
The fundamental idea of the refactoring is to separate the two main functions of JobTracker, resource management and task scheduling/monitoring, into separate components. The new resource manager globally manages the allocation of computing resources for all applications, and the ApplicationMaster of each application is responsible for the corresponding scheduling and coordination. An application is nothing more than a single traditional MapReduce task or a DAG (Directed Acyclic Graph) task. The ResourceManager and each machine's node management server can manage user processes on that machine and organize computations.
In fact, each application's ApplicationMaster is a detailed framework library that combines the resources obtained from the ResourceManager and the NodeManager to work together to run and monitor tasks.
The ResourceManager in the figure above supports hierarchical application queues that share a certain percentage of the cluster's resources. In a sense it is a pure scheduler that does not monitor and track the application during execution. Likewise, it cannot restart tasks that fail due to application failures or hardware errors.
The ResourceManager schedules based on the application's resource requirements; each application requires a different type of resource and therefore requires a different container. Resources include: memory, CPU, disk, network, and more. It can be seen that this is significantly different from the resource usage model of the fixed type of Mapreduce, which has a negative impact on the usage of the cluster. The resource manager provides a plugin for scheduling policies, which is responsible for allocating cluster resources to multiple queues and applications. Scheduling plugins can be based on existing capacity scheduling and fair scheduling models.
The NodeManager in the above figure is the agent of each machine framework, the container that executes the application, monitors the resource usage (CPU, memory, hard disk, network) of the application and reports to the scheduler.
The responsibilities of each application's ApplicationMaster are: ask the scheduler for the appropriate resource container, run tasks, track the status of applications and monitor their progress, and handle the reasons for task failures.
What are the advantages of the Yarn framework over the old MapReduce framework? We can see that:
1) This design greatly reduces the resource consumption of the JobTracker (that is, the current ResourceManager), and makes the program monitoring the status of each Job subtask (tasks) distributed, which is safer and more beautiful.
2) In the new Yarn, ApplicationMaster is a changeable part, users can write their own AppMst for different programming models, so that more types of programming models can run in the Hadoop cluster, you can refer to the official configuration template of hadoop Yarn mapred-site.xml configuration.
3) The representation of resources is based on memory (in the current version of Yarn, CPU usage is not considered), which is more reasonable than the number of remaining slots before.
4) In the old framework, a big burden of JobTracker is to monitor the running status of tasks under the job. Now, this part is thrown to ApplicationMaster, and there is a module in ResourceManager called ApplicationsMasters (note that it is not ApplicationMaster), it is Monitor the health of the ApplicationMaster and restart it on other machines if something goes wrong.
5) Container is a framework proposed by Yarn for resource isolation in the future. This point should be learned from the work of Mesos. Currently, it is a framework that only provides isolation of Java virtual machine memory. The hadoop team's design ideas should support more resource scheduling and control in the future. Since resources are expressed as memory, there is no