What is Mesos? A distributed systems kernel
Mesos is an open source distributed resource management framework under Apache, which is called the kernel of a distributed system. Mesos was originally developed by AMPLab at the University of California, Berkeley, and has since been widely used at Twitter.
Apache Mesos abstracts CPU, memory, storage, and other compute resources away from machines (physical or virtual), enabling fault-tolerant and elastic distributed systems to easily be built and run effectively.
Mesos is built using the same principles as the Linux kernel, only at a different level of abstraction. The Mesos
kernel runs on every machine and provides applications (e.g., Hadoop, Spark, Kafka, Elasticsearch) with API’s for
resource management and scheduling across entire datacenter and cloud environments.
Mesos originated from Borg, Google's data center resource management system. Twitter took inspiration from Google's Borg system and developed a similar resource management system to help them get rid of the dreaded "whale of failure." Then they noticed a project called Mesos under development at UC Berkeley's AMPLab, led by Ben Hindman, a Ph.D. student at UC Berkeley. Later Ben Hindman joined Twitter to develop and deploy Mesos. Now that Mesos manages the deployment of applications on Twitter's more than 30,000 servers, "failure whales" are a thing of the past.
Mesos contains four main types of services (actually a socket server), which are Mesos Master, Mesos Slave, SchedulerProcess and ExecutorProcess. They communicate with each other through Protocol Buffer messages. Several Protocol Buffers are registered inside each service. A message handler, once a certain message is received, it will call the corresponding message handler for processing.
Mesos also adopts the popular Master/Slave master-slave cluster management architecture in distributed clusters. The Mesos master node is the center of the entire cluster. It is responsible for managing and distributing the computing resources of the entire Mesos cluster, scheduling tasks submitted by the upper-level Framework, and managing and distribute the status of all tasks. This master-slave architecture is simple in design and can meet the needs of most normal cluster operations. It still exists in many distributed systems, such as Hadoop and MySQL clusters. However, there is a fatal flaw in this simple design, that is, the Mesos master must continue to exist in the cluster as a service program. Although it is isolated, its status is very important and cannot be lost.
In a cluster with a single Mesos master node, if the Mesos master node fails or the service is unavailable, although the tasks on each Slave node can continue to run, new resources in the cluster will no longer be allocated to the upper-layer Framework, and the upper-layer Framework will It is no longer possible to submit new tasks with an already received offer, and it is not possible to receive status updates for running tasks. In order to solve this problem, improve the high availability of the Mesos cluster, and reduce the impact of the failure of the Mesos master node, the Mesos cluster adopts the traditional active-standby redundancy mode (Active-Standby) to support the deployment of multiple Mesos masters in a Mesos cluster. The node uses ZooKeeper to elect the leader. The elected leader is responsible for sending the cluster resources to each framework in the upper layer in the form of an offer, and processing the requests of the cluster administrator and the upper framework. Monitor the current status. When the Mesos master node goes down or the service is interrupted, the new leader will be quickly selected from the followers to take over all the services, which reduces the downtime of the Mesos cluster service and greatly improves the availability of the cluster.
Kubernetes is an open source project with Google-style cluster management functions for virtual machine or physical machine scenarios. It runs well on modern operating system environments (such as CoreOS and Red Hat Atomic) and provides lightweight compute nodes that you can manage.
Mesos is a cluster manager based on two-phase scheduling. Highly simplified, Mesos manages cluster resources and provides it with a high-level "framework" that accepts these resources to launch tasks. Instead, Kubernetes is a cluster manager based on the idea behind Borg. Google's cluster manager has no concept of two-phase scheduling.
