Background brief
There are two main containerization technologies currently supported by Mesos, one is the Mesos Containerizer technology and the other is the Docker Containerizer technology. Mesos Containerizer mainly uses some features of the operating system itself, such as cgroup, namespace, etc., to isolate containers; Docker Containerizer mainly manages containers by calling Docker API through Mesos Agent.
But there are some problems with managing multiple containerization technologies today:
If you maintain multiple containerization technologies in Mesos, then when adding new functions to container management, you need to make changes to different containerization technologies and implement these functions in different containerization technologies. At least for now, it needs to be implemented separately in Mesos Containerizer and Docker Containerizer.
Docker Containerizer mainly manages Docker containers by calling Docker API, but Docker API has limited opportunities for Mesos customization, especially for container runtime customization, such as some Entrypoint, Cmd, Env, etc., all functions All need to rely on Docker's API; and Mesos Containerizer allows users to customize some Mesos Isolators (isolators) to achieve their own requirements for container isolation, such as some requirements for networking, storage, etc., which can be done by Mesos Isolator. Customization.
Docker Containerizer needs to rely on Docker Daemon. If Docker Daemon hangs and restarts, all Docker Containerizers cannot be restored to their previous state. Mesos needs to ensure that even if the user's Docker Daemon hangs, the user can still create a Docker Container from a Docker image.
Docker is not very stable when it is large-scale. Mesos once discovered a bug and found that some containers could not be stopped normally. Later, after investigation, it was found that the reason was that there were too many Docker containers on the server, which caused the docker daemon to be unable to handle it. reacted. But this situation is not too much, because most of the domestic customers are not very large.
Now there are many specs for containers, including Docker, AppC, OCI, and more. In the future, these different specs may be unified, but now there are many formats, and it is not very convenient for users to use them. If different containerization technologies are currently implemented for different specs, it will be more difficult to maintain in the future.
In fact, I personally feel that there is another reason that the Docker community is not open enough. Docker can now provide enough Offers for customers, including Docker Cloud in the form of public cloud, and Docker Data Center in private cloud mode. It is difficult for other open source software to use Docker profit.
Based on the above reasons, Mesos introduced a new function in 0.28, called unified container management technology, mainly to make some improvements to the current Mesos Containerizer, so that Mesos Containerizer can manage containers of different specs, including Docker, AppC and OCI, so far, has completed support for Docker and AppC, and will complete support for OCI in the future. As for the existing Docker Containerizer, it should be maintained by the Mesos community in the short term, but there should be no major changes.
Principle analysis
The above picture is the overall architecture diagram of Mesos Containerizer. The yellow part is some new modules added to realize the unified container management of Mesos, mainly for the two different specs of Docker and AppC. Different Provision modules are added.
The main function of the Launcher is to start the mesos executor to execute the user's job. The main function of the Provision module is to create the root filesystem for Docker and AppC containers. Isolator mainly does some configuration work for the Mesos Containerizer container, such as some dynamic configuration, some recovery work after Mesos Agent restarts, some cleanup work when the container is destroyed, and obtain some dynamic information when the container is running, etc.
When the container is created, Mesos Containerizer will help the container to create the root file system before the Isolator configures the container. When the container is destroyed, the Mesos Containerizer deletes the container's root filesystem after all the Isolators have cleaned up.
Introduction to Provisioner
The implementation principles of DockerProvisioner and MesosProvisioner are basically the same, including the following steps:
Obtaining a Docker or AppC image now supports two methods: local and remote. Mesos Agent can obtain images from local or remote URIs. Obtaining an image only needs to be obtained locally or remotely when the image is used for the first time. After the acquisition is completed, it will be stored in the local cache. In the future, all containers that use the image will obtain the image from the cache, which improves the container. creation speed. There is a problem with storing containers in the local cache, that is, if the remote image is updated, the newly created container cannot use the updated remote image because the previous image has been cached locally. Mesos is now addressing this issue (https://issues.apache.org/jira/browse/MESOS-4886), adding a "Force image acquisition" flag to the Mesos Image Protobuf, framework developers who need to force image acquisition to To update the local cache, you only need to use this flag to obtain the image from the remote end again to update the local cache.
Create a root filesystem for the container from the obtained Docker or AppC image. Now Mesos mainly supports three ways to create a file system from an image file:
Copy Backend: This is the easiest way, and it is also the default rootfs creation method of Mesos Agent. Copy Backend mainly copies the image file to the root path of the created container to create the root file system of the container. The disadvantage of Copy Backend is that this method is only suitable for relatively small image files, such as 10 to 100M image files. If the image is relatively large, using Copy Backend will have higher IO requirements.
Bind Backend: Bind Backend is mainly designed for single-layer large image files, such as single-layer 5G image files. The reason why Bind Backend is fast is that it only needs to do a Bind operation on the image, basically without any IO. However, Bind Backend also has some shortcomings. For example, it supports single-layer image files, and the file system of Bind Backend is read-only. Users need to mount some readable and writable Volumes for the container of Bind Backend to implement read and write operations. This You can use Mesos Agent to support external storage (click for details).
Overlay Backend: Overlay Backend is mainly to solve the shortcomings of the previous two Backends: Copy Backend has high IO requirements, and Bind Backend can only handle single-layer images. For a specific introduction to Overlay, you can refer to here. Overlay Backend also has two limitations. The first is that Overlay Backend only supports multi-layer images, not single-layer images. The second is that the file system of the Overlay Backend is also read-only, and requires the help of external storage to realize the read and write operations of the container. But Mesos is planning to make some improvements to these two limitations, and make the Overlay Backend support both single-layer and multi-layer images, and the rootfs is also writable. This issue is now being resolved, see here for details.
For the creation of multiple file systems for containers, Mesos' long-term plan is to implement a flexible back-end selection method: Mesos Agent selects the appropriate system creation method according to the format, size, and operating system configuration of the image. For example, if the image is large, Mesos Agent will choose Bind Backend or overlay Backend to speed up the creation of the container's root file system; if the image is not very large, Mesos Agent will choose Copy Backend, etc.
Isolator Improvements
One of the main purposes of the Mesos unified container technology is to realize the customization of the container and some runtime configuration, so the Mesos unified container also introduces some isolators to manage the container. The main ones are three Isolators.
Docker Runtime Isolator
Docker Runtime Isolator is mainly to support some runtime configuration information of docker images, such as Entrypoint, Cmd. Env, WorkingDir and so on. For example, if the user is running the framework's Task, if the Task is not a shell command, then the Docker Runtime Isolator will use Entrypoint or Cmd to run the user's job. For a detailed introduction to Docker Runtime Isolator, you can refer to here.
CNI Isolator
Before the introduction of CNI, Mesos Containerizer used host mode by default for the network of Docker containers. CNI Isolator is mainly used to configure the container network by using some CNI commands of AppC. For example, some CNI commands can be called through some network configuration files, such as bridge, flannel, host-local, etc. to configure the container network. When using this function, the user needs to specify the path of the CNI configuration file and the path of the CNI command to ensure that the Mesos Agent can call the CNI command according to the configuration file to configure the container network when using the CNI Isolator. The main part of this project has been completed, you can refer to here and here for details.
Docker Volume Isolator
The main function of this isolator is to enable Mesos to use external storage through the Docker Volume Driver API. After the integration is complete, Mesos can use all the external storage resources supported by the Docker Volume Driver to store some of the user's job running results. This project is divided into two stages. The first stage is to use a dvdcli developed by EMC to obtain the mount point of external storage through dvdcli. Therefore, when users use this function, they need to specify the path of dvdcli when the agent starts, so that Mesos Agent calls dvdcli to get the mount point of the container. The second stage is to get rid of the dependence on dvdcli and use the logic of dvdcli as a C++ library of Mesos. Docker Volume Isolator can directly obtain the mount point by calling this C++ library. The first phase of the project has now been completed, users can use this function by installing dvdcli. Details can be found here and here.
Advantages of Unified Container Management Technology
The advantages of the Mesos unified container management technology are mainly to solve some of the problems in the "Background Introduction":
It simplifies the development and maintenance of Mesos through one containerization technology to manage all containers of different specs, such as Docker, AppC, etc.
Get rid of the dependence on Docker Daemon, if you use Mesos Containerizer, even if Docker Daemon hangs, users can still create Docker containers through Mesos Containerizer.
Mesos Containerizer allows users to customize some Mesos Isolators (isolators) to achieve their own requirements for container isolation, such as some requirements for network, storage, etc., which can be customized through Mesos Isolator.
Follow-up
AppC improvements
AppC does not currently have a Runtime Isolator similar to Docker, which results in that when using AppC images, some runtime configurations of Appc cannot be used, such as environment variables, exec, working directory, etc. This issue is being improved, please refer to here for details.
Another problem is AppC's image discovery mechanism. Currently AppC only supports a very simple image discovery mechanism. When users start the Mesos Agent, they need to specify the discovery mechanism of the AppC image through the parameter appc_simple_discovery_uri_prefix. For example user can use http://, https://, hdfs://:9000/user/abc/cde, file:///tmp/appc/ etc. The default is http://, then Mesos Agent The image name will be added after the URI to complete the URI. The format of the image name is {name}-{version}-{os}-{arch}.{ext}. For example, a user's appc_simple_discovery_uri_prefix is configured as file:///tmp/appc/, and then when the user starts the task, the specified AppC image is ubuntu, the user's os is Linux, and the arch is amd64, then the final Mesos Agent will be The AppC image will be found from file:///tmp/appc/ubuntu-latest-linux-amd64.aci. Now AppC does not support the mirror discovery mode of meta, and JIRA is already tracking this issue, please refer to here and here for details.
OCI support
At present, Mesos Containerizer only supports Docker and part of AppC for unified container management, and the support for OCI has not yet begun. One of the main reasons is that the standard of OCI has not yet been finalized, so this piece is not yet a priority for Mesos. very high. Mesos now has JIRA tracking this issue, details.
Mesos Container CLI
At present, Mesos does not have commands similar to "docker ps", "docker inspect", etc. to view the status of all containers. In the future, Mesos plans to introduce similar commands to help users easily view the status of all Mesos Containers in the system through the mesos command, etc. .
test steps
Because all the functions of the unified container management technology are implemented in Mesos Agent, and all functions are controlled by parameters, now we mainly introduce some configurations and parameters and the startup test command of the Agent.
Configuration parameters
If a user wants to use the functionality of the Mesos unified container, it is important to have several Agent parameters.
1. –isolation: Mainly configure the isolator used by the current Mesos Agent. For example, if you use the Docker container through Mesos Containerizier, you must configure docker/runtime as the isolator, otherwise the Agent cannot be started.
2. --image_providers: Configure the current Agent's image to provide this. Now only APPC and DOCKER are supported.
3. –appc_simple_discovery_uri_prefix: Configure the image prefix of APPC. This parameter is described in “Follow-up work”.
4. --docker_registry: Configure Docker's registry, now supports Docker Hub, Local Registry and local path.
Docker test
In order to speed up the test, I use the local mirror test method. First, store some Docker images locally. You can use the command "docker save" to store the images locally. Taking busybox as an example, you can use the command "docker save -o busybox:latest.tar busybox" to store the busybox images locally.
root@mesos002:~# ls /root/reglocal
alpine.tar busybox:latest.tar ubuntu:14.04.tar
Mesos Agent startup command:
GLOG_v=1 ./bin/mesos-slave.sh --master=192.168.56.12:5050 --isolation=docker/runtime,filesystem/linux --image_providers=docker --launcher=linux --executor_environment_variables="{}" --docker_registry=/root/reglocal
The test uses the test command that comes with Mesos:
./src/mesos-execute --master=192.168.56.12:5050 --name=test_mesos --docker_image=busybox:latest --containerizer=mesos --command="sleep 10"
I0404 01:08:42.160414 14128 scheduler.cpp:172] Version: 0.29.0
Subscribed with ID '3327bb81-f2a0-400a-8411-79910af7b343-0001
task test_mesos submitted to agent 030ca997-9310-48bb-973f-d53136022537-S0
Received status update TASK_RUNNING for task test_mesos
Received status update TASK_FINISHED for task test_mesos
When the command is running, you can view the Mesos Agent log to understand the Mesos Agent workflow. From the Mesos Agent log at this time, you can see how the Mesos Agent obtains the image, how to create the root file system, and so on.
AppC test
Also in order to speed up the test, I use the local mirror test method. First store some AppC images locally. There are many ways to create an AppC image. The easiest way is to convert a Docker image into an AppC image by commanding docker2aci. For details, refer to here.
root@mesos002:~# ls /root/appclocal/
ubuntu-latest-linux-amd64.aci ubuntu-latest.aci ubuntu2-latest-linux-amd64.aci
Mesos Agent startup command:
GLOG_v=1 ./bin/mesos-slave.sh --master=192.168.56.12:5050 --executor_registration_timeout=5mins --containerizers=mesos --isolation=filesystem/linux --image_providers=APPC --appc_simple_discovery_uri_prefix=/root/appclocal/
The test uses the test command that comes with Mesos:
./src/mesos-execute --master=192.168.56.12:5050 --command="/bin/echo ello ubuntu;" --name=test --appc_image=ubuntu
I0404 01:12:35.006664 14223 scheduler.cpp:172] Version: 0.29.0
Subscribed with ID '3327bb81-f2a0-400a-8411-79910af7b343-0002
task test submitted to agent 030ca997-9310-48bb-973f-d53136022537-S0
Received status update TASK_RUNNING for task test
Received status update TASK_FINISHED for task test
http://www.chinacloud.cn/show.aspx?id=23455&cid=12