7K words take you to understand Docker technology in simple terms

what is docker

Docker is an open source application container engine that allows developers to package their applications and dependencies into a portable image, and then publish it to any popular Linux or Windows operating system machine, and can also implement virtualization. Containers use a sandbox mechanism completely, and there will be no interfaces between them.

container technology

Containers decouple software applications from the operating system, giving users a clean and minimal Linux environment while running everything else in one or more isolated "containers." The purpose of containers is to start a limited set of applications or services (often called microservices) and have them run in an isolated sandbox environment.

This isolation prevents a process running in a given container from monitoring or affecting a process running in another container. Again, these containerized services do not affect or interfere with the host machine. The idea of ​​being able to consolidate many services scattered across multiple physical servers into one is one of the many reasons data centers opt for this technology.

Containers vs Virtual Machines

First, let's look at the following picture, the following is the implementation of the virtual machine

The way for virtual machines to achieve resource isolation is to run an independent slave operating system on the main operating system. As shown in the figure above, the server that we know is running at the bottom layer, and usually a main operating system will run on the server.

The Guest OS of the virtual machine is the operating system installed on the virtual machine , which is a complete operating system kernel; the Hypervisor layer of the virtual machine can be simply understood as a hardware virtualization platform, which is responsible for coordinating the allocation and management of hardware resources on the host. A more classic virtual machine software is Parallels Desktop.

Let's take a look at the implementation of Docker

The implementation of Docker container technology is one layer less than the implementation of virtual machine technology. Since Docker does not need Hypervisor to realize hardware resource virtualization, programs running on Docker containers directly use the hardware resources of actual physical machines . Therefore, Docker will have an advantage in efficiency in terms of CPU and memory utilization.

Finally, do a simple comparison.

characteristic	container	virtual machine
start up	second level	minute level
Hard disk usage	Usually MB	Generally GB
performance	close to native	weaker than
System Support	A single machine supports thousands of containers	Usually dozens

Use of Docker

There may be some readers who have not used docker directly. Let’s give an example first. In the absence of docker before, if we want to run a mysql in a linux environment, we may first download the compressed package, decompress it, and compile it. , setting and other processes are very complicated.

But when we use docker to run mysql, we only need to run it first docker pull mysqland it will automatically download the latest mysql image.

After running it docker run mysqlagain, you can run it directly. If you want to run another mysql instance, you only need to run docker runthe command again and set a new port to run it. So you can see if docker is very useful.

The functions of docker are far more than these. Next, let's walk into docker and see how docker implements container technology.

Docker basic concepts

run process

In the flowchart of Docker running, we can simply understand the image as an executable program, and the Container is the running process. Registry is the code management platform.

Then writing a program requires source code, then the "source code" that "writes" an image is the dockerfile, and docker is the "compiler".

Therefore, we only need to specify in the dockerfile which programs are needed and what configurations to rely on, and then hand the dockerfile to the "compiler" docker for "compilation" to generate an "executable program" image, and then we can run this image, image After running it is Docker container.

Image(mirror image)

A Docker image is a special file system. In addition to providing the programs, libraries, resources, configuration and other files required by the container runtime, it also contains some configuration parameters prepared for runtime (such as anonymous volumes, environment variables, users, etc. ). Images do not contain any dynamic data, and their contents are not changed after they are built.

Let's look at a simpler example first:

The picture above is a resume sample with the debian system as the basic image. You can see that the middle layer is the basic image. We have not performed any customized operations on the image. After running, a container is generated, and the container is the one that can be used the object to write.

For Linux, after the kernel starts, the root file system will be mounted to provide user space support for it. The Docker image (Image) is equivalent to a root file system.

Of course, the functions that Docker can achieve are far more than that. Let's take a look at how to use DockerFile to build a customized image:

sql

copy code

FROM debian RUN apt-get install emacs RUN apt-get install apache2 CMD ["/bin/bash"]

When designing Docker, it makes full use of Union FS technology and designs it as a hierarchical storage architecture . Mirroring is actually composed of multi-layer file systems.

When the image is built, it will be built layer by layer, and the previous layer is the basis of the next layer. After each layer is built, it will not change again, and any changes on the next layer only occur on its own layer.

The feature of hierarchical storage also makes the reuse and customization of images easier. You can even use the previously built image as the base layer, and then further add new layers to customize what you need and build a new image.

But extra attention should be paid when building the image. For example, the operation of deleting the file of the previous layer does not actually delete the file of the previous layer, but only marks the file as deleted in the current layer. When the final container is running, although this file will not be seen, in fact, the file will always follow the image. Therefore, when building a mirror, each layer should try to contain only the things that need to be added to the layer, and any extra things should be cleaned up before the end of the layer construction.

Container(container)

The definition of a container  is almost the same as that (container) of an image  (image) , the only difference is that the top layer of the container is readable and writable.

In a broad sense, we can understand the container as, container = image + read-write layer.

Repository(storehouse)

After the image is built, it can be easily run on the current host, but how to run the image on other servers, then we need a place to store image files centrally. This leads to the concept Docker Repositoryof

Docker Registry (Warehouse Registration Server) is a centralized storage and distribution image service. Docker The concept of a warehouse is  Git similar, and the registration server can be understood as  GitHub such a hosting service. In fact, a  Docker Registry warehouse can contain multiple warehouses  (Repository) , and each warehouse can contain multiple labels  (Tag), and each label corresponds to an image. Therefore, the mirror warehouse is  Docker a place for centralized storage of mirror files, similar to the code warehouse we used before.

Usually, a warehouse will contain images of different versions of the same software , and tags are often used to correspond to each version of the software  . We can use <仓库名>:<标签>the format to specify which version of the software is the mirror image. If no label is given, . will be used  latest as the default label.

Warehouses can be divided into two types:

• public(public warehouse)
• private(private warehouse)

Docker Registry The public warehouse is a service that is open to users and allows users to manage images  Registry . Generally, such public services allow users to upload and download public images for free, and may provide paid services for users to manage private images.

In addition to using public services, users can also build private services locally  Docker Registry . Docker The official image is provided  Docker Registry, which can be directly used as a private  Registry service. After the user creates his own image, he can use  push the command to upload it to the public or private warehouse, so that the next time he uses the image on another machine, he only needs to  pull download it from the warehouse.

Docker architecture

Docker is a C/S model architecture, the backend is a loosely coupled architecture, and each module performs its duties.

1. All commands of the user pass through Docker Clientthe Docker Daemonestablishment of communication with and send requests to the latter.
2. Docker DaemonAs the main part of the Docker architecture, it first provides the functions of the Server so that it can accept Docker Clientrequests;
3. Engine executes a series of jobs inside Docker, and each job exists in the form of a Job.

In the previous basic concepts, we have learned about Registry, Container and other concepts. Next is the introduction of some components during Docker operation.

Docker Client

1. Docker ClientIs Docker Daemonthe client to establish communication with. After using the docker command, the user Docker Clientis responsible for parsing the corresponding command and parameters, and Docker Daemoninitiating a request to the server.
2. Docker ClientDocker DaemonCommunication can be established in the following three ways :
   1. tcp://host:port
   2. unix://path_to_socket
   3. fd://socketfd
3. Docker ClientAfter the container management request is sent, Docker Daemonthe request is accepted and processed. When Docker Clientthe returned request is received and processed, Docker Clienta complete life cycle is over. A complete request: send a request → process the request → return the result, which is consistent with the traditional C/S architecture request process.

Docker Daemon

Docker Daemon is the daemon process of docker and the core of docker runtime. It consists of two parts.

1. Docker Server
   1. Docker ServerIt is equivalent to the server side of the C/S architecture. The function is to accept and dispatch requests sent by Docker Client. After accepting the request, the Server finds the corresponding Handler to execute the request through routing and distribution scheduling.
   2. During the server's service process, the server accepts Docker Clientthe access request on the listener and creates a brand new goroutine to serve the request. In the goroutine, first read the request content, then do the parsing work, then find the corresponding routing item, then call the corresponding Handler to process the request, and finally the Handler replies to the request after processing the request.
2. Engine
   1. EngineIt is the running engine in the Docker architecture, which Jobmanages all containers and images through execution.
   2. In Enginethe process of data structure design and implementation, there is a handler object. The handler object stores all the handler handling access for many specific jobs. For example, Engineif there is an item in the handler object: {“create”: daemon.ContainerCreate,}, it means that when the job named "create" is running, it is executed daemon.ContainerCreate的handler.
3. Job
   1. One Jobcan be considered as Enginethe most basic work execution unit inside the Docker architecture. Every job that Docker can do, can be abstracted into one Job. Whether it is the download of the image, the stop of the container, and so on. Docker ServerThe running process is actually one Job, named serveapi.
   2. JobThe concept is similar to the process in Unix. In the Unix process, there are names, parameters, environment variables, standard input and output, error handling, return status, etc. for each process, which also Jobexist in Docker.

Graph

Graph manages all locally downloaded images. Graph DB records the dependencies between all images.

Driver

Through the Driver driver, Docker can realize the customization of the Docker container operating environment. The dimensions of customization mainly include the network environment, storage method, and container execution method.

The implementation of Docker Driver can be divided into the following three types of drivers: graphdriver, networkdriver and execdriver.

graphdriver

Graphdriver is mainly used to complete the management of container images, including storage and retrieval.

graphdriver is mainly used for container image management:

1. Responsible for downloading images from Docker Registry and storing them. When a user downloads a specified container image, graphdriver stores the container image hierarchically in a specified local directory.
2. Responsible for obtaining the specified container image from the local image storage directory, and preparing rootfs for the container according to specific rules;
3. Responsible for managing brand new images built from the specified Dockerfile.

networkdriver

The role of networkdriver is to complete the configuration of the Docker container network environment, including:

1. Create a bridge for the Docker environment when the Docker Daemon starts;
2. Allocate corresponding network interface resources to the Docker container before it is created;
3. Docker container assigns IP, port and performs NAT port mapping with the host machine, sets container firewall policy, etc.

execdriver

As the execution driver of the Docker container, execdriver is responsible for creating the namespace of the container runtime, managing the statistics and restrictions on the use of container resources, and responsible for the actual operation of the internal process of the container, etc.

Before Docker version 0.9.0, only the LXC driver of Linux was supported for container management. After version 0.9.0, the native driver was used by default. The native driver is a new sub-project of the docker project, which removes external dependencies.

libcontainer

ibcontainer is a library designed and implemented in the Go language in the Docker architecture. The original intention of the design is to hope that the library can directly access container-related system calls in the kernel without relying on any dependencies.

It is precisely because of the existence of libcontainer that docker can complete operations on firewalls, namespaces, etc. without relying on LXC or other packages.

Docker Container

Docker Container (Docker container) is the ultimate embodiment of service delivery in the Docker architecture. Docker manages through DockerDaemon, executes libcontainer, and finally creates a Docker container. As a delivery unit, a Docker container is similar in function to a traditional virtual machine (Virtual Machine), with the characteristics of limited resources and isolation of the environment from the outside world.

Process combing

Seeing this, I believe that readers and friends should already have a general understanding of the Docker infrastructure. Let us take docker run as an example to review how the various components work together  Docker .

Suppose we want to run a:docker run -p 3306:3306 --name mysql -d mysql

The container startup process is as follows:

• Docker The client executes  docker run the command
• Docker daemonPull the latest   mirror  by graghdrivergoing toGraghmysql
• Docker daemon By networkdriverestablishing port mapping
• Docker daemonexecdriverStart the container  by

Docker core technology implementation

After learning about so many Docker implementations, we may have some final questions about how Docker achieves resource isolation.

When using Linux or macOS on a daily basis, we don't need to run multiple completely separate servers, but if we start multiple services on the server, these services will actually affect each other, and each service can see other services The process of the host machine can also access any file on the host machine. This is something we don't want to see in many cases. We hope that different services running on the same machine can be completely isolated, just like running on multiple different servers . same as on the machine.

Docker uses a total of three technologies to achieve complete isolation between containers :

Namespaces(Namespaces)

Namespaces (namespaces) are a method provided by Linux for us to separate resources such as process trees, network interfaces, mount points, and inter-process communication.

process isolation

In Docker Deamonthe early stage of startup, setNamespacesa new namespace will be created through the function. clonePassing CLONE_NEWPIDparameters in the function used to create the namespace completes the process isolation between the container and the host.

File Resource Isolation

clonePass parameters in the function used to create the namespace CLONE_NEWNS, and the child process can get a copy of the parent process mount.

When a container is created, the container needs its own rootfs to achieve isolation from other container file resources, so when Docker creates a container, it will mount the directory required by the container, and change the root directory that the container can access. file system isolation between them.

pivot_root Change the root node that the container can access a file directory, or  chroot functions provided by libcontaine .

network isolation

When Docker starts, it will automatically create a docker0 virtual network bridge on the host, which is actually a bridge of Linux, which can be understood as a software switch. It will forward between the network ports mounted to it.

When creating a Docker container, a pair of veth pair interfaces will be created at the same time. One end of this pair of interfaces is inside the container, which is eth0 inside the container;

A veth pair is a virtual network device interface that appears in pairs. One end is connected to the network protocol stack, and the other end is connected to each other. The connected ends communicate with each other.

The other end is local and mounted to the docker0 bridge with a name starting with veth. In this way, the host can communicate with the container, and the containers can communicate with each other. Docker creates a virtual shared network between the host and all containers.

Control Groups(control group)

Control Groups (CGroups for short) can isolate the physical resources on the host machine, and CGroup provides the following functions.

• Limit the number of resources a process group can use (Resource limiting). For example, the memory subsystem can set a memory usage limit for the process group. Once the memory used by the process group reaches the limit and then applies for memory, OOM (out of memory) will be triggered.

• Process group priority control (Prioritization). For example: you can use the cpu subsystem to assign a specific cpu share to a process group.

• Record the amount of resources used by the process group (Accounting). For example: you can use the cpuacct subsystem to record the cpu time used by a process group

• Process group isolation (Isolation). For example: using the ns subsystem, different process groups can use different namespaces to achieve the purpose of isolation. Different process groups have their own process, network, and file system mount spaces.

• Process group control (Control). For example: use the freezer subsystem to suspend and resume process groups

physical resource isolation

CGroup controls the allocation of system resources through multiple subsystems.

We can use lssubsys -mto view the subsystem directory corresponding to CGroup under the current system.

• cpuset /sys/fs/cgroup/cpuset
• cpu,cpuacct /sys/fs/cgroup/cpu,cpuacct
• blkio /sys/fs/cgroup/blkio
• memory /sys/fs/cgroup/memory
• devices /sys/fs/cgroup/devices
• freezer /sys/fs/cgroup/freezer
• net_cls,net_prio /sys/fs/cgroup/net_cls,net_prio
• perf_event /sys/fs/cgroup/perf_event
• pids /sys/fs/cgroup/pids

In the host machine, first, when Docker starts, the docker folder will be created under all the above subsystems.

When Docker creates a container, it will create a new file corresponding to the pid in the task subdirectory under the docker folder to allocate and control container resources.

UnionFS(union file system)

Union file system (UnionFS) is a layered, lightweight and high-performance file system, which supports the modification of the file system as a submission to superimpose layer by layer, and can mount different directories to the same virtual under the file system

In Docker, each image layer is built on top of another image layer, and all image layers are read-only. Only the topmost container layer of each container can be directly read and written by the user, so you need A file system manages all files.

Mirror management

In Docker, a variety of file systems are currently used to manage images, including the current mainstream overlay2, aufsand so on.

Different storage drivers also have completely different implementations when storing images and container files. Interested readers can  find the corresponding content in Docker's official document Docker storage drivers .

Summarize

Seeing this, I believe that readers and friends have a deeper understanding of Docker.

Since Docker has been updated so far, the code base is too large, and we can only get a glimpse of it from the low version of Docker source code and the Docker technical articles of the big guys. If you are interested, you can also communicate with each other.