Container Resource Visibility Using the LXCFS File System

Foreword:

Linux Cgrouputilizes limit container resources, but when you run topthe command in the container, you will find that the information displayed is the CPU and memory data of the host machine, not the data of the current container. The reason for this problem is that /procthe file system does not understand Cgroupthe existence of restrictions.

A common practice in the community is to leverage lxcfsthe filesystem to provide resource visibility to containers.

1. Basic introduction

---

LXCFS is an open source Fuse (user mode file system) that supports LXC containers and Docker containers. After startup, a file with the same name as the file in /procthe directory , so as to ensure that the container reads the information data in lxcfsthe maintained /procfile when reading data.

• LXCFS provides the procfsfollowing .

/proc/cpuinfo
/proc/diskstats
/proc/meminfo
/proc/stat
/proc/swaps
/proc/uptime

The schematic diagram of LXCFS is as follows:

• For example, after mounting the host's /var/lib/lxcfs/proc/meminfofile to proc/meminfothe location of the container;
• When the process in the container reads the content of the corresponding file, the implementationLXCFS of will read the correct memory limit from the corresponding to the container.FuseCgroup

Two, LXCFS installation and use

---

1. Install the LXCFS file system

[root@localhost ~]# wget http://copr-be.cloud.fedoraproject.org/results/ganto/lxc3/epel-7-x86_64/01041891-lxcfs/lxcfs-3.1.2-0.2.el7.x86_64.rpm   
[root@localhost ~]# yum -y localinstall lxcfs-3.1.2-0.2.el7.x86_64.rpm
[root@localhost ~]# sed -i 's@ExecStart=.*@ExecStart=/usr/bin/lxcfs -o nonempty /var/lib/lxcfs/@g' /usr/lib/systemd/system/lxcfs.service
[root@localhost ~]# systemctl daemon-reload
[root@localhost ~]# systemctl start lxcfs && systemctl enable lxcfs

If the following error occurs when lxcfsrestarting , it means that the mounted directory is not empty, and -o nonemptythe parameter needs to be added.

• fuse: mountpoint is not empty
• fuse: if you are sure this is safe, use the 'nonempty' mount option

2. Implement container resource visibility based on Docker

[root@localhost ~]# docker run -it -m 512Mb --cpus 2 \
-v /var/lib/lxcfs/proc/cpuinfo:/proc/cpuinfo:rw \
-v /var/lib/lxcfs/proc/diskstats:/proc/diskstats:rw \
-v /var/lib/lxcfs/proc/meminfo:/proc/meminfo:rw \
-v /var/lib/lxcfs/proc/stat:/proc/stat:rw \
-v /var/lib/lxcfs/proc/swaps:/proc/swaps:rw \
-v /var/lib/lxcfs/proc/uptime:/proc/uptime:rw \
tomcat:8.5.87 /bin/bash

3. Implement container resource visibility based on Kubernetes

• Inject the settings through https://github.com/denverdino/lxcfs-admission-webhookthe .Admission WebhookLXCFS
• Need to check api-versionswhether started admissionregistration.k8s.io/v1beta1(1.9.0+)

kubectl api-versions | grep 'admissionregistration.k8s.io/v1beta1'

1) Install lxcfsthe file system

[root@localhost ~]# git clone https://github.com/denverdino/lxcfs-admission-webhook.git
[root@localhost ~]# cd lxcfs-admission-webhook
[root@localhost lxcfs-admission-webhook]# sed -i '/^metadata/a\  namespace: kube-system' deployment/lxcfs-daemonset.yaml  
[root@localhost lxcfs-admission-webhook]# kubectl apply -f deployment/lxcfs-daemonset.yaml

---

2) Modify the Kube-APIServer configuration file and enable the service plug-in

• Increase:MutatingAdmissionWebhook,ValidatingAdmissionWebhook

[root@localhost ~]# vim /etc/systemd/system/kube-apiserver.service
--enable-admission-plugins=

3) Restart the Kube-APIServer service

[root@localhost ~]# systemctl daemon-reload
[root@localhost ~]# systemctl restart kube-apiserver

4) Install lxcfs-admission-webhookthe injector service

[root@localhost lxcfs-admission-webhook]# bash deployment/install.sh
creating certs in tmpdir /tmp/tmp.kK3cUmQvdG
Generating RSA private key, 2048 bit long modulus
............+++
...................+++
e is 65537 (0x10001)
certificatesigningrequest.certificates.k8s.io/lxcfs-admission-webhook-svc.default created
NAME                                  AGE   REQUESTOR   CONDITION
lxcfs-admission-webhook-svc.default   1s    admin       Pending
certificatesigningrequest.certificates.k8s.io/lxcfs-admission-webhook-svc.default approved
secret/lxcfs-admission-webhook-certs created
NAME                            TYPE     DATA   AGE
lxcfs-admission-webhook-certs   Opaque   2      0s
deployment.apps/lxcfs-admission-webhook-deployment created
service/lxcfs-admission-webhook-svc created
mutatingwebhookconfiguration.admissionregistration.k8s.io/mutating-lxcfs-admission-webhook-cfg created

Check

[root@localhost lxcfs-admission-webhook]# kubectl get secrets,pods,svc,mutatingwebhookconfigurations

6) Verify

[root@localhost lxcfs-admission-webhook]# kubectl label namespace default lxcfs-admission-webhook=enabled
[root@localhost lxcfs-admission-webhook]# kubectl apply -f deployment/web.yaml

• kubectl label namespace default lxcfs-admission-webhook=enabled: Register all Pods under defaultthe namespace ;
• Here we need to pay attention that the registered namespace should be distinguished from the namespace where lxcfsthe service is located , otherwise the following error will be output after lxcfsbeing restarted:

caused \\\"not a directory\\\"\"": unknown: Are you trying to mount a directory onto a file (or vice-versa)? Check if the specified host path exists and is the expected type

---

[root@localhost lxcfs-admission-webhook]# kubectl exec -it web-7f4dfcc4f4-88rb7 -- /bin/bash
root@web-7f4dfcc4f4-88rb7:/usr/local/apache2# free -h
             total       used       free     shared    buffers     cached
Mem:          256M       6.2M       249M         0B         0B       300K
-/+ buffers/cache:       5.9M       250M
Swap:           0B         0B         0