# Lxcfs资源视图隔离 ## 1. 资源视图隔离 容器中的执行`top`、`free`等命令展示出来的CPU，内存等信息是从`/proc`目录中的相关文件里读取出来的。而容器并没有对`/proc`，`/sys`等文件系统做隔离，因此容器中读取出来的CPU和内存的信息是宿主机的信息，与容器实际分配和限制的资源量不同。 ```bash /proc/cpuinfo /proc/diskstats /proc/meminfo /proc/stat /proc/swaps /proc/uptime ``` 为了实现让容器内部的资源视图更像虚拟机，使得应用程序可以拿到真实的CPU和内存信息，就需要通过文件挂载的方式将cgroup的真实的容器资源信息挂载到容器内`/proc`下的文件，使得容器内执行top、free等命令时可以拿到真实的CPU和内存信息。 ## 2. Lxcfs简介 lxcfs是一个FUSE文件系统，使得Linux容器的文件系统更像虚拟机。lxcfs是一个常驻进程运行在宿主机上，从而来自动维护宿主机cgroup中容器的真实资源信息与容器内`/proc`下文件的映射关系。 lxcfs的命令信息如下： ```bash #/usr/local/bin/lxcfs -h Usage: lxcfs [-f|-d] -u -l -n [-p pidfile] mountpoint -f running foreground by default; -d enable debug output -l use loadavg -u no swap Default pidfile is /run/lxcfs.pid lxcfs -h ``` lxcfs的源码： ## 3. Lxcfs原理 lxcfs实现的基本原理是通过文件挂载的方式，把cgroup中容器相关的信息读取出来，存储到lxcfs相关的目录下，并将相关目录映射到容器内的/proc目录下，从而使得容器内执行top,free等命令时拿到的/proc下的数据是真实的cgroup分配给容器的CPU和内存数据。 **原理图** ![lxcfs](https://res.cloudinary.com/dqxtn0ick/image/upload/v1626002975/article/kubernetes/resource/lxcfs/lxcfs.webp) **映射目录** | 类别 | 容器内目录 | 宿主机lxcfs目录 | | --- | ------------------------------ | ------------------------------------------------------------ | | cpu | /proc/cpuinfo | /var/lib/lxcfs/{container\_id}/proc/cpuinfo | | 内存 | /proc/meminfo | /var/lib/lxcfs/{container\_id}/proc/meminfo | | | /proc/diskstats | /var/lib/lxcfs/{container\_id}/proc/diskstats | | | /proc/stat | /var/lib/lxcfs/{container\_id}/proc/stat | | | /proc/swaps | /var/lib/lxcfs/{container\_id}/proc/swaps | | | /proc/uptime | /var/lib/lxcfs/{container\_id}/proc/uptime | | | /proc/loadavg | /var/lib/lxcfs/{container\_id}/proc/loadavg | | | /sys/devices/system/cpu/online | /var/lib/lxcfs/{container\_id}/sys/devices/system/cpu/online | ## 4. 使用方式 ### 4.1. 安装lxcfs 环境准备 ```bash yum install -y fuse fuse-lib fuse-devel ``` 源码编译安装 ```bash git clone git://github.com/lxc/lxcfs cd lxcfs ./bootstrap.sh ./configure make make install ``` 或者通过rpm包安装 ```bash wget https://copr-be.cloud.fedoraproject.org/results/ganto/lxc3/epel-7-x86_64/01041891-lxcfs/lxcfs-3.1.2-0.2.el7.x86_64.rpm; rpm -ivh lxcfs-3.1.2-0.2.el7.x86_64.rpm --force --nodeps ``` 查看是否安装成功 ```bash lxcfs -h ``` ### 4.2. 运行lxcfs 运行lxcfs主要执行两条命令。 ```bash sudo mkdir -p /var/lib/lxcfs sudo lxcfs /var/lib/lxcfs ``` 可以通过systemd运行。 lxcfs.service文件： ```bash cat > /usr/lib/systemd/system/lxcfs.service < ### 5.1. lxcfs-image [Dockerfile](https://github.com/denverdino/lxcfs-admission-webhook/blob/master/lxcfs-image/Dockerfile) ```bash FROM centos:7 as build RUN yum -y update RUN yum -y install fuse-devel pam-devel wget install gcc automake autoconf libtool make ENV LXCFS_VERSION 3.1.2 RUN wget https://linuxcontainers.org/downloads/lxcfs/lxcfs-$LXCFS_VERSION.tar.gz && \ mkdir /lxcfs && tar xzvf lxcfs-$LXCFS_VERSION.tar.gz -C /lxcfs --strip-components=1 && \ cd /lxcfs && ./configure && make FROM centos:7 STOPSIGNAL SIGINT COPY --from=build /lxcfs/lxcfs /usr/local/bin/lxcfs COPY --from=build /lxcfs/.libs/liblxcfs.so /usr/local/lib/lxcfs/liblxcfs.so COPY --from=build /lxcfs/lxcfs /lxcfs/lxcfs COPY --from=build /lxcfs/.libs/liblxcfs.so /lxcfs/liblxcfs.so COPY --from=build /usr/lib64/libfuse.so.2.9.2 /usr/lib64/libfuse.so.2.9.2 COPY --from=build /usr/lib64/libulockmgr.so.1.0.1 /usr/lib64/libulockmgr.so.1.0.1 RUN ln -s /usr/lib64/libfuse.so.2.9.2 /usr/lib64/libfuse.so.2 && \ ln -s /usr/lib64/libulockmgr.so.1.0.1 /usr/lib64/libulockmgr.so.1 COPY start.sh / CMD ["/start.sh"] ``` star.sh ```bash #!/bin/bash # Cleanup nsenter -m/proc/1/ns/mnt fusermount -u /var/lib/lxcfs 2> /dev/null || true nsenter -m/proc/1/ns/mnt [ -L /etc/mtab ] || \ sed -i "/^lxcfs \/var\/lib\/lxcfs fuse.lxcfs/d" /etc/mtab # Prepare mkdir -p /usr/local/lib/lxcfs /var/lib/lxcfs # Update lxcfs cp -f /lxcfs/lxcfs /usr/local/bin/lxcfs cp -f /lxcfs/liblxcfs.so /usr/local/lib/lxcfs/liblxcfs.so # Mount exec nsenter -m/proc/1/ns/mnt /usr/local/bin/lxcfs /var/lib/lxcfs/ ``` ### 5.2. daemonset [lxcfs-daemonset.yaml](https://github.com/denverdino/lxcfs-admission-webhook/blob/master/deployment/lxcfs-daemonset.yaml) ```yaml apiVersion: apps/v1 kind: DaemonSet metadata: name: lxcfs labels: app: lxcfs spec: selector: matchLabels: app: lxcfs template: metadata: labels: app: lxcfs spec: hostPID: true tolerations: - key: node-role.kubernetes.io/master effect: NoSchedule containers: - name: lxcfs image: registry.cn-hangzhou.aliyuncs.com/denverdino/lxcfs:3.1.2 imagePullPolicy: Always securityContext: privileged: true volumeMounts: - name: cgroup mountPath: /sys/fs/cgroup - name: lxcfs mountPath: /var/lib/lxcfs mountPropagation: Bidirectional - name: usr-local mountPath: /usr/local volumes: - name: cgroup hostPath: path: /sys/fs/cgroup - name: usr-local hostPath: path: /usr/local - name: lxcfs hostPath: path: /var/lib/lxcfs type: DirectoryOrCreate ``` ### 5.3. [lxcfs-admission-webhook](https://github.com/denverdino/lxcfs-admission-webhook) `lxcfs-admission-webhook`实现了一个动态的准入webhook，更准确的讲是实现了一个[修改性质的webhook](https://kubernetes.io/zh/docs/reference/access-authn-authz/admission-controllers/#mutatingadmissionwebhook)，即监听pod的创建，然后对pod执行patch的操作，从而将lxcfs与容器内的目录映射关系植入到pod创建的yaml中从而实现自动挂载。 [deployment](https://github.com/denverdino/lxcfs-admission-webhook/blob/master/deployment/deployment.yaml) ```yaml apiVersion: apps/v1 kind: Deployment metadata: name: lxcfs-admission-webhook-deployment labels: app: lxcfs-admission-webhook spec: replicas: 1 selector: matchLabels: app: lxcfs-admission-webhook template: metadata: labels: app: lxcfs-admission-webhook spec: containers: - name: lxcfs-admission-webhook image: registry.cn-hangzhou.aliyuncs.com/denverdino/lxcfs-admission-webhook:v1 imagePullPolicy: IfNotPresent args: - -tlsCertFile=/etc/webhook/certs/cert.pem - -tlsKeyFile=/etc/webhook/certs/key.pem - -alsologtostderr - -v=4 - 2>&1 volumeMounts: - name: webhook-certs mountPath: /etc/webhook/certs readOnly: true volumes: - name: webhook-certs secret: secretName: lxcfs-admission-webhook-certs ``` 具体部署参考:[install.sh](https://github.com/denverdino/lxcfs-admission-webhook/blob/master/deployment/install.sh) ```bash #!/bin/bash ./deployment/webhook-create-signed-cert.sh kubectl get secret lxcfs-admission-webhook-certs kubectl create -f deployment/deployment.yaml kubectl create -f deployment/service.yaml cat ./deployment/mutatingwebhook.yaml | ./deployment/webhook-patch-ca-bundle.sh > ./deployment/mutatingwebhook-ca-bundle.yaml kubectl create -f deployment/mutatingwebhook-ca-bundle.yaml ``` 执行命令 ```bash /deployment/install.sh ``` 参考： * * * *