Author: Zen and the Art of Computer Programming
1 Introduction
Mesos is a distributed system resource manager framework open sourced by the Apache Foundation. It has now become one of the basic components of Hadoop, Spark and Aurora. Mesos itself is an independently running cluster management system that can provide job scheduling, resource isolation, fault tolerance and other functions. In addition, it includes a distributed file system (HDFS) called MESOS-SANDBOX, which supports shared data access between tasks across hosts. Therefore, Mesos can be understood as a high-performance distributed computing framework that combines the two most famous open source projects, HDFS file system and Apache YARN resource management system. Mesos is designed from the perspective of low-level developers, with a modular architecture and good scalability. Its main features are as follows:
1) Distributed multi-process architecture: Mesos uses a distributed multi-process architecture model to achieve horizontal expansion of functions.
2) Robust fault-tolerance mechanism: Mesos can automatically perform fault-tolerance processing when encountering various faults to ensure service availability.
3) Support multiple programming languages: Mesos provides APIs for multiple programming languages such as Java, C++, Python, Go, etc., making it easier for application developers to integrate Mesos and build their own systems.
4) Reliable scheduling policy: Mesos uses a policy-based priority scheduling algorithm to ensure that tasks are distributed as evenly as possible throughout the cluster.
The source code analysis of Mesos is designed to help readers understand the overall architecture of Mesos and learn the functional logic implemented by its architecture. This article will introduce each module of Mesos and its implementation details one by one. I hope that by reading this article, readers can have a deeper understanding and application of Mesos.
2.Mesos module
Mesos consists of the following modules:
1) Master process: Responsible for managing all resources in the cluster and scheduling applications.
2) Agent process: Responsible for executing tasks on each node.
3) Executor process: A command that runs on each node and is responsible for running application code.
4) Scheduler process: Responsible for tracking resource availability, task progress and status.
5) Framework process: Mesos provides a framework that allows users to write their own schedulers to meet specific business needs.
3.Master process
3.1 Roles and responsibilities
The Master process has two roles:
1) Master: Elected when the cluster starts, it is used to coordinate the cluster work and ensure the overall stable operation of the cluster. When the main Master hangs up, the Slave on another machine will act as the new main Master.
2) From Master: Synchronize task information when the cluster starts and use it for task scheduling. When the primary Master fails, the secondary Master will take over and continue task scheduling.
The main responsibilities of the Master are as follows:
1) Resource management: Master adjusts task scheduling and the order of resource allocation by checking the utilization and idleness of resources;
2) Task scheduling: Master decides how to deploy resources and execute tasks for applications based on task requirements, cluster resource status and task history;
3) Fault-tolerant recovery: The Master is responsible for monitoring the health status of each component in the cluster, and if an abnormality is found, it will self-repair or notify the corresponding component;
4) Configuration management: Master provides a dynamic configuration center that can update application configurations in real time and notify various components;
5) Log management: Master can record logs of all events in the cluster to facilitate problem tracing.
3.2 Function overview
Master plays an important role in the entire Mesos. Its main functions are as follows:
1) Resource management: The Master uniformly manages the resources in the cluster and schedules and allocates them based on resource utilization and idleness.
2) Task scheduling: Master selects the appropriate Slave to execute the task based on the task priority, hardware requirements, network bandwidth and other constraints.
3) Fault-tolerant recovery: The Master has self-healing capabilities and can quickly detect, diagnose and solve problems when failures occur, keeping the cluster highly available.
4) Configuration management: Master provides a dynamic configuration center. Each component subscribes to configuration items of interest as needed and obtains the latest version in real time.
5) Log management: Master records logs of all events in the cluster, including component startup, stop, crash, failure and other behaviors, making it easier for administrators to locate and troubleshoot problems in a timely manner.
3.3 Master startup process
The startup process of Mesos Master is shown in the figure below:
The figure describes the Master startup process in detail. When the Master process starts, it first registers itself with the ZooKeeper service and waits to be elected as the master. After that, it will wait for the Master election to complete, and then perform resource management, task scheduling and fault-tolerant recovery of the entire cluster.
Since the Master needs to manage the resources of the entire cluster, when the Master starts, it first needs to read the configuration file and perform initialization settings according to the parameters in the configuration file and the cluster environment. The Master will read the Slaves information, available resources, etc. in the current cluster, and perform capacity calculations to determine the resources available on each Slave in the cluster.
The Master will send this information to all Slaves through heartbeats. Afterwards, the Master will establish communication channels with all Slaves and enable heartbeat monitoring. If a slave does not respond to heartbeats for a period of time, the master will consider the slave to be unavailable, and the master will remove the slave from the cluster.
When the Master receives a request from a Slave, the Master will allocate resources to the Slave. The Master determines which Slave should allocate resources to perform tasks by evaluating the priority of tasks on different Slaves, the idleness of hardware resources, network bandwidth and other factors. The Master will also write task information to the database for persistent storage.
When the Slave starts executing a task, the Master checks whether the Slave is ready to accept the task. If the Slave is unavailable or has insufficient resources, the Master will notify other Slaves to perform tasks instead. When the slave completes the task, the master will update the status of the task and reallocate resources until all tasks are completed.
4.Agent process
4.1 Roles and responsibilities
The Agent process is mainly responsible for executing applications and is generally started on each node. When the Agent starts, it will report its existence to the Master and obtain task information in the cluster. The Agent will send heartbeat packets to the Master regularly. The Master will consider the Agent to be working normally only after receiving the Agent's heartbeat.
4.2 Function overview
Agent plays an important role in the entire Mesos. Its main functions are as follows:
1) Execute tasks: Agent receives tasks sent by Master and starts an Executor process locally to run the task code.
2) Resource management: The Agent obtains the resources allocated by the Master and schedules them according to the idle status of the resources and the task execution progress.
3) Fault-tolerant recovery: Agent has self-healing capabilities and can quickly detect, diagnose and solve problems when a fault occurs, and resume normal work as soon as possible.
4) Log management: Agent records the log of each task, including task start, stop, failure and other behaviors, to facilitate administrators to locate and troubleshoot problems in a timely manner.
4.3 Agent startup process
The startup process of Mesos Agent is shown in the figure below:
The figure describes the Agent startup process in detail. When the Agent process starts, it will report its existence to the Master and wait for the Master to send tasks. If the Master detects a Slave request to start a new task, the Agent will download the relevant dependencies and images, and then start the Executor process to run the task code.
The Agent sends messages to the Master through heartbeats and receives instructions from the Master at any time. When the Agent cannot send heartbeats to the Master normally for some reasons, or is disconnected for a long time for some reason and exceeds a certain time limit, the Master will consider the Agent unavailable, and the Agent will ask other Slaves for task resources.
When the Agent receives a task from the Master, the Agent will start an Executor process to run the task code. The Executor process is a user-defined command line program that is mainly responsible for executing specified tasks. The Agent will establish a pipeline with the Executor process and transmit task-related information through the pipeline, such as commands, input and output files, execution environment, etc.
When the Executor process completes the task, the Agent will feed back the results to the Master and continue to execute the next task according to the Master's instructions.
5.Framework process
Mesos also provides a framework for developing user-defined schedulers. Framework is an independent process that runs on a user-defined computing framework. Framework initiates various requests by calling the interface provided by Master, such as submitting tasks, declaring resources, querying task status, etc. Framework can make full use of Mesos's resource scheduling, fault tolerance and elastic expansion capabilities to improve resource utilization, reduce computing costs, and meet performance requirements in specific application scenarios.
6. Summary and outlook
The main purpose of this Mesos source code analysis is to make readers more familiar with the overall architecture of Mesos and the implementation principles of each module of Mesos. Readers can further understand the internal operating mechanism of Mesos by reading the source code.
This article only introduces some core modules of Mesos, and there are many important modules and features that are not mentioned. In subsequent articles, we will introduce other aspects of Mesos and discuss the key design ideas and implementation principles.