[Container] Docker security and log management

The root cause of container security issues is that the container and the host share the kernel. If an application in a container causes the Linux kernel to crash, the entire system may crash. Unlike a virtual machine, a virtual machine does not share the kernel with the host. A virtual machine crash will generally not cause the host to crash.

The difference between Docker containers and virtual machines

1. Isolation and Sharing
   By adding a Hypervisor layer (virtualization middle layer), the virtual machine virtualizes virtual hardware such as network cards, memory, and CPU, and then creates a virtual machine on top of it. Each virtual machine has its own system kernel.
   The Docker container uses isolation to isolate file systems, processes, devices, networks and other resources, and then controls permissions, CPU resources, etc., ultimately making the containers inseparable from each other. Impact, the container cannot affect the host.
   The container and the host share resources such as the kernel, file system, and hardware.

2. Performance and Consumption
   Compared with virtual machines, container resource consumption is less. Under the same host, more containers can be created than virtual machines.
   However, the security of virtual machines is slightly better than that of containers. To break into the host or other virtual machines from the virtual machine, you need to break through the Hypervisor layer first, which is extremely difficult.
   The docker container shares resources such as the kernel and file system with the host, and is more likely to affect other containers and the host.

tie	Docker container	virtual machine
Startup speed	Quick, a few seconds	Slow, a few minutes
Running performance	Close to native (runs directly in the kernel)	Running on Hypervisor, about 50% loss
Disk usage	Small, even dozens of KB (depending on the image layer)	Very large, up to GB
Concurrency	A host can start hundreds or thousands of containers	Up to dozens of virtual machines
Isolation	process level	System level (more thorough)
operating system	Mainly supports Linux	Almost all
Encapsulation level	Only package project code and dependencies, sharing the host kernel	Complete operating system, isolated from the host machine

Security issues with Docker

1. Docker’s own vulnerabilities
As an application, Docker itself will have code flaws in its implementation. CVE official records There are more than 20 vulnerabilities in the historical versions of Docker, which can be found on the Docker official website.
Common attack methods used by hackers mainly include code execution, privilege escalation, information leakage, permission bypass, etc. Currently, Docker versions change very quickly, and Docker users can upgrade Docker to the latest version.

2. Docker source code issues
Docker provides Docker hub, which allows users to upload created images for other users to download and quickly build an environment. But it also brings some safety issues.
For example, the following three methods:
(1) Hackers upload malicious images
If a hacker implants them in the created image Trojans, backdoors and other malware, then the environment is already unsafe from the beginning, and there will be no security at all in the future.

(2) The image uses vulnerable software
Among the images that can be downloaded on DockerHub, 75% of the images have vulnerable software installed. Therefore, after downloading the image, you need to check the version information of the software inside to see if there are vulnerabilities in the corresponding version, and update and patch it in a timely manner.

(3) Man-in-the-middle attack to tamper with the image
The image may be tampered with during the transmission process. The current new version of Docker has provided a corresponding verification mechanism to prevent this problem.

Docker architectural flaws and security mechanisms

The architecture and mechanism of Docker itself may cause problems. For example, in an attack scenario, a hacker has controlled some containers on the host, or gained access to a container by establishing it on the public cloud, and then launched an attack on the host or other containers. attack.

1. LAN attacks between containers
   Containers on the host can form a LAN, so ARP spoofing, port scanning, broadcast storm and other attack methods on the LAN can be used.
   Therefore, deploying multiple containers on a host requires proper network security configuration, such as setting iptables rules.

2. DDoS attack exhausts resources
   The Cgroups security mechanism is to prevent such attacks. This problem can be avoided by not allocating too many resources to a single container.

3. Vulnerable system call
   An important difference between Docker and virtual machines is that Docker and the host share the same operating system kernel.
   Once there is a vulnerability in the host kernel that allows privileges to be exceeded or escalated, even though Docker is executed as a normal user, when the container is invaded, the attacker can also use the kernel vulnerability to jump to the host to do more things.

4. Shared root user permissions
   If you run the container with root user permissions (docker run --privileged), the root user in the container also has the root permissions of the host.

Docker Security Baseline Standard

The following summarizes Docker security baseline standards from six aspects: kernel, host, network, mirror, container and others.

1. Kernel level
   (1) Update the kernel in time.
   (2) User NameSpace (the root authority within the container is in a non-high authority state outside the container).
   (3) Cgroups (quotas and measurements of resources), set resource limits such as CPU, memory, disk IO, etc.
   (4) Add additional security by properly hardening the system by enabling SELinux (controlling file access permissions).
   (5) Capability (permission division), such as allocating designated CPUs to containers.
   (6) Seccomp (limited system calls), restrict unnecessary system calls.
   (7) It is forbidden to share the container's namespace with the host process namespace, such as host network mode.

2. Host level
   (1) Create an independent partition for the container, such as on a distributed file system.
   (2) Only run necessary services, and try to avoid running ssh services in containers.
   (3) It is forbidden to map sensitive directories on the host to containers. You need to pay attention when creating data volumes with -v.
   (4) Audit the Docker daemon process, related files and directories to prevent the generation of viruses or Trojan files.
   (5) Set the appropriate default number of file descriptors.
   (6) The access permissions of Docker-related files with user permissions of root should be 644 or lower.
   (7) Periodically check the container list of each host and clean up unnecessary containers.

3. Network level
   (1) Use iptables to set rules to prohibit or allow network traffic between containers.
   (2) Allow Docker to modify iptables.
   (3) It is forbidden to bind Docker to other used IP/Port or Unix Socket.
   (4) It is forbidden to map privileged ports on the container.
   (5) Only the required ports are opened on the container.
   (6) It is forbidden to use the host network mode on the container.
   (7) If the host has multiple network cards, bind the container incoming traffic to a specific host network card.
   docker network create --subnet=172.18.0.0/16 --opt "com.docker.network.bridge.name"="docker1" mynetwork
   docker run -itd --net mynetwork --ip 172.18.0.100 centos:7 /bin/bash
   iptables -t nat -A POSTROUTING -s 172.18.0.100 -o ens36 -j SNAT --to -source 192.168.80.10

4. Image level
   (1) Create a local private image warehouse server.
   (2) The software in the image is the latest version. It is recommended to use the corresponding version according to the actual situation. Business stability is given priority.
   (3) Use trusted image files and download them through secure channels.
   (4) Rebuild the image instead of patching the container and image, destroy the abnormal container and rebuild it.
   (5) Reasonably manage image tags and promptly remove images that are no longer used.
   (6) Use mirror scanning.
   (7) Use image signature.

5. Container level
   (1) Containers are minimized and the minimum set of operating system images is used.
   (2) The container runs as a single main process.
   (3) Disable the --privileged flag from using privileged containers.
   (4) It is forbidden to run the ssh service on the container. Try to use docker exec to enter the container.
   (5) Mount the root directory system of the container in a read-only manner, -v host directory:container directory:ro.
   (6) Clearly define the data drive letter belonging to the container.
   (7) Limit the number of times the container attempts to restart by setting on-failure. Repeated restarts of the container are prone to data loss, –restart=on-failure:3.
   (8) Limit the number of processes available in the container, docker run -m limits memory usage to prevent fork bombs. (fork bomb, rapidly growing child processes, exhausting the number of system processes).(){.|.&};.

6. Other settings
   (1) Regularly conduct security audits of the host system and containers.
   (2) Run the container using the least resources and the least permissions. This is the core idea of ​​Docker container security.
   (3) Avoid deploying a large number of containers on the same host and maintain a manageable number.
   (4) Monitor the usage, performance and other indicators of Docker containers, such as zabbix.
   (5) Add real-time threat detection and event alarm response functions, such as zabbix.
   (6) Use central and remote log collection services, such as ELK.

Since security is a very specific technology, I won’t go into details here. You can directly refer to Docker’s official documentation, https://docs.docker.com/engine/security/

Common security configuration methods related to containers

1. Minimize the container
   If only necessary services are run in the container, services such as SSH cannot be easily opened to connect to the container. The following methods are usually used to enter the container.

docker exec -it a661258f6bfe bash

2. Docker remote API access control
   Docker's remote call API interface has an unauthorized access vulnerability, and at least external network access should be restricted. It is recommended to use Socket method for access.

//在 docker 服务配置文件指定监听内网 ip
vim /usr/lib/systemd/system/docker.service
--13行--修改
ExecStart=/usr/bin
/dockerd -H unix:///var/run/docker.sock -H tcp://192.168.80.10:2375

（2）重启 Docker
systemctl daemon-reload
systemctl restart docker
netstat -natp | grep 2375

（3）在宿主机的 firewalld 上做 IP 访问控制，source address 指定的是客户端地址
firewall-cmd --permanent --add-rich-rule="rule family="ipv4" source address="192.168.80.15" port protocol="tcp" port="2375" accept"
firewall-cmd --reload
或
iptables -t filter -A INPUT -s 192.168.80.15 -p tcp --dport 2375 -j ACCEPT 

（4）在客户端上实现远程授权访问
docker -H tcp://192.168.80.10 images

Limit traffic flow

//Use firewall filters to limit the source IP address range of the Docker container from communicating with the outside world.
firewall-cmd --permanent --zone=public --add-rich-rule="rule family="ipv4" source address="192.168.80.0/24" reject"
iptables -t filter -A INPUT -s 192.168.80.0/24 -j REJECT

//A large number of problems in the production environment are caused by vulnerabilities caused by the exposure of Docker container ports. In addition to problems with operating system account permission control, there are also hidden dangers in the process management of Docker Daemon.
Currently, the commonly used Docker versions support Docker Daemon to manage host iptables, and once the process is started and the port mapping of -p host_port:guest_port is added, Docker Daemon will directly add the corresponding FORWARD Chain and - j ACCEPT, and the default DROP rule is made in the INPUT chain, which cannot restrict docker, which leaves serious security risks. Therefore, it is recommended:
(1) Do not use the Docker service on a machine with an external network IP.

(2) Use docker orchestration systems such as k8s to manage Docker containers.

(3) Add --iptables=false to the Docker daemon startup command on the host, then write the commonly used iptables rules into the file, and then use iptables-restore to redirect the input to refresh the rules.

Image security

In general, to ensure that images are only obtained from trusted repositories, it is recommended to use the harbor private repository.
If the company does not use its own image source, it needs to use the Docker image security scanning tool Clair to check the downloaded image. By scanning the image synchronously with the CVE database, verifying the md5 and other feature values ​​of the image, and then further building based on the image after confirming consistency. Once a vulnerability is discovered, the user will be notified to deal with it, or the image will be directly prevented from continuing to be built.

Avoid information leakage in Docker containers

In recent years, a large number of personal or corporate account passwords have been leaked on Github. When such problems occur, dockerfile or docker-compose files are generally used to create containers. If there is authentication information such as account and password in these files, once the Docker container is opened to the outside world, sensitive information on these hosts will also be leaked.

Communication security between DockerClient and DockerDaemon

In order to prevent link hijacking, session hijacking and other problems that lead to man-in-the-middle attacks during Docker communication, both ends of the c/s should communicate through TLS encryption.

By creating a TLS key certificate on the server and then issuing it to the client, the client accesses the container through the private key, thus ensuring the security of docker communication.
//Workflow for access using certificates:
(1) The client initiates an HTTPS request and connects to port 443 of the server.
(2) The server must first apply for a set of digital certificates (certificate contents include public key, certificate authority, expiration date, etc.).
(3) The server sends its own digital certificate to the client (the public key is in the certificate, and the private key is held by the server).
(4) After receiving the digital certificate, the client will first verify the validity of the certificate. If the certificate verification passes, a pseudo-random number generator (/dev/random) will be used to randomly generate a [symmetric key], and the public key of the certificate will be used to encrypt the [symmetric key].
(5) The client sends the public key encrypted [symmetric key] to the server.
(6) After receiving the ciphertext key from the client, the server uses its previously reserved private key to asymmetrically decrypt it. After decryption, the client's [symmetric key] is obtained. ], and then use the client's [symmetric key] to encrypt the returned data, so that the transmitted data is all ciphertext.
(7) The server returns the encrypted ciphertext data to the client.
(8) After receiving it, the client uses its own [symmetric key] to symmetrically decrypt it and obtain the data returned by the server.

First create the ca certificate, which is just an officially certified certificate. Next, create the certificates for the server and client nodes.
There are three steps to create a certificate at this time:
(1) Set the private key to ensure safe encryption
(2) Use the private key Sign to ensure that the identity is authentic and non-repudiation
(3) Use the ca certificate to make the certificate

master 192.168.80.10 docker-ce-cli-20.10.5-3.el7.x86_64 docker-ce
client 192.168.80.15 docker-ce-cli-20.10.5- 3.el7.x86_64 docker-ce
Note: Since the go version used by the 20.10.9 version of the docker client is go1.16.8, and go1.15 and later versions do not support private CA generation certificate, so the docker client here still uses the version installed by docker-ce-cli-20.10.5-3.el7.x86_64.

//首先创建一个存放目录
mkdir /tls
cd /tls/

//Generate ca certificate
(1) Create ca private key
openssl genrsa -aes256 -out ca-key.pem 4096 #Input 123123

genrsa: Use RSA algorithm to generate private key
-aes256: Use AES algorithm with 256-bit key to encrypt the private key, so that every time you use the private key file, you will enter the password , can be omitted
-out: The path of the output file, if no output file is specified, it is the standard output
4096: Specify the length of the private key, the default is 1024. This item must be the last parameter on the command line

(2) Create ca certificate
openssl req -new -x509 -days 1000 -key ca-key.pem -sha256 -subj “/CN=*” -out ca. pem

req: Execute certificate issuance command
-new: New certificate issuance request
-x509: Generate x509 format certificate, specially used when creating a private CA Use
-days: the validity period of the certificate in days
-key: specify the private key path
-sha256: certificate The summary uses the sha256 algorithm
-subj: user information related to the certificate (abbreviation of subject)
-out: path to the output file

//Use ca certificate to issue server-side certificate
(3) Create server private key
openssl genrsa -out server-key.pem 4096< /span>

(4) Generate certificate signing request file (csr file)
openssl req -new -key server-key.pem -sha256 -subj “/CN=*” -out server .csr

(5) Use the ca certificate and private key certificate to issue the server signing certificate, enter 123123, (signature request file, ca certificate, ca key required)
openssl x509 -req -sha256 -in server.csr -CA ca.pem -CAkey ca-key.pem -CAcreateserial -days 1000 -out server-cert.pem

x509: generate x509 format certificate
-req: input csr file
-in: csr file to be input
-CA: Specify the path of the ca certificate
-CAkey: Specify the private key path of the ca certificate
-CAcreateserial: Indicates the creation of the certificate serial number file, created The default name of the serial number file is ca.srl

//用ca证证书签发client端证书
（6）生成客户端私钥
openssl genrsa -out client-key.pem 4096

（7）生成证书签名请求文件
openssl req -new -key client-key.pem -subj "/CN=client" -out client.csr

（8）创建扩展配置文件，使秘钥适合客户端身份验证
echo extendedKeyUsage=clientAuth > extfile.cnf

（9）使用 ca 证书签发客户端签名证书，输入 123123，（需要签名请求文件，ca 证书，ca 密钥）
openssl x509 -req -sha256 -in client.csr -CA ca.pem -CAkey ca-key.pem -CAcreateserial -extfile extfile.cnf -days 1000 -out client-cert.pem 

//删除两个证书签名请求文件和扩展配置文件
rm -rf ca.srl client.csr extfile.cnf server.csr

//配置docker服务配置文件
vim /lib/systemd/system/docker.service
--13行--修改
ExecStart=/usr/bin/dockerd --tlsverify --tlscacert=/tls/ca.pem --tlscert=/tls/server-cert.pem --tlskey=/tls/server-key.pem -H tcp://0.0.0.0:2376 -H unix:///var/run/docker.sock

//重启docker服务
systemctl daemon-reload
systemctl restart docker
netstat -natp | grep 2376
setenforce 0

//将 /tls 目录中的 ca.pem、client-cert.pem、client-key.pem 三个文件复制到另一台主机
scp ca.pem [email protected]:/etc/docker/
scp client-cert.pem [email protected]:/etc/docker/
scp client-key.pem [email protected]:/etc/docker/


//本地验证，使用证书访问时要用主机名连接，docker 不支持使用 IP 进行证书访问
hostnamectl set-hostname master
su

vim /etc/hosts	
127.0.0.1   master

docker --tlsverify --tlscacert=ca.pem --tlscert=server-cert.pem --tlskey=server-key.pem -H tcp://master:2376 images


//在客户端上操作
hostnamectl set-hostname client
su

vim /etc/hosts
192.168.80.10   master

cd /etc/docker/
docker --tlsverify --tlscacert=ca.pem --tlscert=client-cert.pem --tlskey=client-key.pem -H tcp://master:2376 images