Containerization and microservices are currently the hottest topics. Not long ago, the author (it is said that the title of "author" is no longer appropriate because the pen is no longer needed, because the keyboard is used for input, it is more appropriate to be called "key person") to participate in QCon A microservice monitoring session in Shanghai was full of scenes. I had to listen to the entire session in the crowded aisle. As there are more and more containers to be managed, a cluster management platform for containers has become a necessity!
Docker Swarm
Swarm is a new container cluster management tool released by Docker in early December 2014. It can turn multiple hosts into one virtual Docker host to manage. Swarm is developed in Go language and is open source, and its full source code can be found on github . Swarm uses standard Docker APIs to bring Docker users a seamless cluster experience. In July 2016, Swarm has been integrated into Docker Engine.
Function
Docker Swarm provides API and CLI to manage clusters running Docker, and its functionality is not fundamentally different from using local Docker. But you can bring good scalability by adding Node. In theory, you can have an infinite Docker host by adding Nodes.
Swarm does not provide UI, if you need UI, you can install UCP , but unfortunately, this UCP is charged.
Architecture
The architecture of Swarm is not complicated, it can be said to be very simple. Manager is responsible for container scheduling, Node is responsible for container operation, Node runs Docker Daemon and Manager communicates through HTTP. The Docker Client uses the standard Docker API exposed on the Manager to use Docker's functionality.
Swarm's cluster coordination and business discovery can support different third-party components, including:
- Consul
- Etcd
- ZooKeeper
- Docker Hub
If you are not familiar with the concept of cluster coordination, you can refer to my other blog "Using Python for Distributed System Coordination (ZooKeeper, Consul, etcd)"
There are three basic container scheduling strategies for Swarm:
- spread containers as far as possible on different nodes
- binpack containers are distributed on the same node as much as possible
- random containers are distributed on random nodes
The high availability configuration of a Swarm cluster is easy, as shown in the following figure:
The managers are configured in different AWS AZs, and the primary manager is elected through leader election.
Multiple nodes are distributed in different AZs, and Consul/etcd/ZooKeeper also need to be redundant.
Features
Docker Swarm is characterized by simple configuration and architecture. Using Docker's native API, it can integrate well with the Docker ecosystem.
Governors
Kubernetes is an open source container application management system developed by Google to manage the deployment, maintenance and expansion of applications. Using Kubernetes makes it easy to manage containerized applications running across machines. Kubernetes is also developed in Go language and the source code can be found on github .
Kubernetes originated from Google's internal container management system Borg. After years of experience in the production environment, it is very powerful.
Function
Kubernetes mainly provides the following functions:
- Use Docker to package, instantiate, and run applications.
- Run and manage containers across machines as a cluster.
- Solve the communication problem between Docker cross-machine containers.
- Kubernetes' self-healing mechanism makes container clusters always run in the state expected by users.
- High Availability and Reliable Scaling of Applications
- Support online upgrade of applications (Rolling Update)
- Supports deployment across cloud platforms (IaaS)
In order to support these functions, Kubernetes has made a lot of abstract concepts. Therefore, when you start using Kubernetes, you need to learn a lot of new concepts, including:
- Pod
Pod is the basic operation unit of Kubernetes. One or more related containers are formed into a Pod. Usually, the containers in the Pod run the same application or related applications. The containers contained in the Pod run on the same Minion (Host), which is regarded as a unified management unit and share the same volumes and network namespace/IP and Port space. - Job
Job is an application with a relatively short life cycle. Generally, it will only restart in the case of an error. You can expand the job by configuring the concurrency and running times of the job. - Service
Service is an application with a long life cycle and will restart at any exit. High expansion and high availability can be achieved by configuring Service replication (Replica). - Recplica Controller The
Replication Controller ensures that the specified number of pod replicas (replicas) are running in the Kubernetes cluster at any time. If there are less than the specified number of pod replicas (replicas), the Replication Controller will start a new Container, otherwise it will kill the excess to ensure The quantity does not change. Replication Controller creates pods using pre-defined pod templates. Once created successfully, the pod template has nothing to do with the created pods. You can modify the pod template without affecting the created pods, or you can directly update the pods created by Replication Controller.
The above are some core concepts. In addition to these, Kubernetes also provides other concepts to support the operation and maintenance of applications, including:
- Label
manages objects in the system through Label - Namespace
groups objects, resources, and can be used to support multi-tenancy - Config Map
provides global configuration data storage
In short, it has powerful functions, many system concepts, and is relatively complex.
Kubernetes supports the installation of UI addons to manage the entire system:
Architecture
The following figure is the basic architecture of Kubernetes:
- The Master
Master defines the main declaration of the Kubernetes cluster Master/API Server, and the Client (Kubectl) calls the Kubernetes API to manage the entry of the main components of Kubernetes, Pods, Services, Minions, and containers. Master consists of API Server, Scheduler, and Registry.
The Scheduler collects and analyzes the resource (memory, CPU) load of all Minion nodes in the current Kubernetes cluster, and then distributes the newly created Pods to the available nodes in the Kubernetes cluster accordingly. Because once the resources of Minion nodes are allocated to Pods, these resources cannot be allocated to other Pods unless these Pods are deleted or exited. Therefore, Kubernetes needs to analyze the resource usage of all Minions in the cluster to ensure that the distributed workload does not It will exceed the range of available resources of the current Minion node - Minion
Minion is responsible for running Pod, Service, Jobs, and Minion communicates with Master through Kubelet. Proxy solves the problem that the external network can access the application services provided by the containers in the cross-machine cluster.
- etcd
is responsible for cluster coordination and service discovery
Features
Kubernetes provides a lot of application-level management capabilities, including high availability and high scalability. Of course, in order to support these functions, its architecture and concepts are relatively complex. Of course, I think in order to obtain these functions, it is worth it!
Apache Mesos
Mesos is an operating system for software-defined data centers, in which resources across data centers are managed uniformly. The original intention of Mesos was not to manage containers, but with the development of containers, Mesos added the function of containers. Mesos can manage the computing resources of different machines in a unified manner, just like the same operating system for running distributed applications.
Mesos originated from Borg, Google's data center resource management system. You can learn more about the origin of the Borg and its impact on Mesos in this article from WIRED Magazine .
Function
Key features of Mesos include:
- Highly scalable and highly available
- Customizable two-level scheduling
- Provide API for application extension
- Cross-platform
The following figure is the management interface of Mesos:
Architecture
The basic architecture of Mesos is as follows
- Master
Master is responsible for the unified coordination of resources and the management of Slave. The Master coordinates all the slaves, determines the available resources of each node, aggregates and calculates the reports of all available resources across the nodes, and then issues resource invitations to the Framework registered to the Master (as the client of the Master). Mesos implements a two-level scheduling architecture, which can manage many types of applications. The first level of scheduling is the Master daemon, which manages the Slave daemons running on all nodes in the Mesos cluster. A cluster consists of physical or virtual servers that run tasks for applications, such as Hadoop and MPI jobs. The second level of scheduling consists of "components" called Frameworks. - Slave
Salve runs the Executor process to run tasks. - Framework
Framework includes scheduler (Scheduler) and executor (Executor) processes, in which each node will run the executor. Mesos can communicate with different types of Frameworks, each of which is managed by a corresponding application cluster.
The Framework can choose to accept or reject resource offers from the master according to the needs of the application. Once the invitation is accepted, the Master coordinates the Framework and Slave, schedules tasks on participating nodes, and executes them in the container, so that various types of tasks can be executed. - ZooKeeper
Zookeeper is responsible for cluster coordination, Master leader election, etc.
Features
Mesos is more mature than Kubernetes and Swarm, but Mesos mainly solves the abstraction at the operating system level, not specifically designed for containers. If users go out of containers, they need to integrate other applications, such as Hadoop, Spark, Kafka Wait, Mesos is more appropriate. Mesos is a more heavyweight cluster management platform with richer functions. Of course, many functions are based on various Frameworks.
The scalability of Mesos is very good, with a maximum support of 50,000 nodes. If the scalability requirements are very high, Mesos is the best choice.
AWS ECS
ECS (Amazon EC2 Container Service) is a highly scalable, high-performance container cluster service developed by Amazon. Easily run container applications and services on managed clusters of Amazon EC2 instances. His biggest advantage is that on the cloud, there is no need to manage the machines and networks in the data center.
Function
ECS inherits the high scalability and high availability of AWS services, and security can also be guaranteed.
On the basis of basic container management, ECS uses the concepts of Task and Service to manage applications.
Task is similar to Docker Compose, using a JSON to describe the application to run. Service is a higher-level abstraction, including running instances of multiple tasks. By modifying the number of task instances, the scaling of the service can be controlled. At the same time, Service can ensure that the specified number of tasks are running, and when an error occurs, restart the failed task
Architecture
The following figure is the architecture diagram of ECS:
Using the familiar AWS concepts such as EC2, ELB, security groups, etc., AWS users can easily manage your container clusters. And you don't need to pay any fees other than the basic resources at the beginning.
The ECS agent can be a Container to connect to the ECS cluster. ECS Agent is developed using Go and has been open sourced.
We don't know the scheduling policy of ECS, but AWS provides an example if the scheduling policy of a third party is inherited.
Through Cloud Watch Log, we can easily monitor the entire cluster.
Features
If you are a heavy user of AWS, ECS is a good choice, because you can run your container cluster on the AWS cloud, which is very convenient to manage.
Compare
Here is a simple comparison of several platforms:
| Swarm | Governors | Months | ECS | |
| basic skills | Docker's native cluster management tool | An open source container cluster tool that provides application-level deployment, maintenance and expansion capabilities | data center based operating system | Cloud-based highly available, highly scalable container cluster |
| high level abstraction | without | Pod |
without | Task Service |
| Application Extensibility | without | support | Framework support | support |
| Application high availability | without | support | Framework support | support |
| Cluster coordination and service discovery | etcd |
etcd | ZooKeeper | ／ |
| Scheduling strategy (Schedule) | Built-in, extensible | built-in | Two levels, scalable | Scalable |
| monitor | Logging Driver | Heapter，ELK Addon | built-in | CloudWatch |
| User Interface | CLI |
CLI API UI |
API UI |
CLI |
| Development language | Go | Go | Java | NA |
| open source | Yes | Yes | Yes | no |
| Maximum number of supported management nodes | Official 1000 |
Official 1000 |
10000 | ／ |
Summarize
Of the four platforms, Swarm is the lightest and simplest in function, suitable for users with a large number of Docker instance environments. Kubernetes adds many application-level functions, which are suitable for rapid application deployment and maintenance. Mesos is the heaviest and is suitable for environments that already have a considerable mix of applications and containers. ECS is recommended for AWS users or cloud users who don't want to manage their own data centers.