COS将耗尽/ var / lib / docker卷的inode
我试图让我们在GCP上运行一些服务。
我目前看到的一个问题是，我很快启动的虚拟机似乎用完了/ var / lib / docker文件系统的inode。 我以为这是在容器优化的操作系统中调整的东西之一？
wouter@nbwm-cron ~ $ df -hi Filesystem Inodes IUsed IFree IUse% Mounted on /dev/root 78K 13K 65K 17% / devtmpfs 463K 204 463K 1% /dev tmpfs 464K 1 464K 1% /dev/shm tmpfs 464K 500 463K 1% /run tmpfs 464K 13 464K 1% /sys/fs/cgroup tmpfs 464K 9 464K 1% /mnt/disks tmpfs 464K 16K 448K 4% /tmp /dev/sda8 4.0K 11 4.0K 1% /usr/share/oem /dev/sda1 1013K 998K 15K 99% /var tmpfs 464K 45 464K 1% /var/lib/cloud overlayfs 464K 39 464K 1% /etc
wouter@nbwm-cron ~ $ docker images REPOSITORY TAG IMAGE ID CREATED SIZE <name>/stackdriver-agent latest 0c4b075e7550 3 days ago 1.423 GB <none> <none> 96d027d3feea 4 days ago 905.2 MB gcr.io/<project>/nbwm-ops/docker-php5 latest 5d2c59c7dd7a 2 weeks ago 1.788 GB
nbwm-cron wouter # tune2fs -l /dev/sda1 tune2fs 1.43.3 (04-Sep-2016) Filesystem volume name: STATE Last mounted on: /var Filesystem UUID: ca44779b-ffd5-405a-bd3e-528071b45f73 Filesystem magic number: 0xEF53 Filesystem revision #: 1 (dynamic) Filesystem features: has_journal ext_attr resize_inode dir_index filetype needs_recovery extent 64bit flex_bg sparse_super large_file huge_file uninit_bg dir_nlink extra_isize Filesystem flags: signed_directory_hash Default mount options: user_xattr acl Filesystem state: clean Errors behavior: Remount read-only Filesystem OS type: Linux Inode count: 1036320 Block count: 4158971 Reserved block count: 0 Free blocks: 4062454 Free inodes: 1030756 First block: 0 Block size: 4096 Fragment size: 4096 Group descriptor size: 64 Reserved GDT blocks: 747 Blocks per group: 32768 Fragments per group: 32768 Inodes per group: 8160 Inode blocks per group: 510 Flex block group size: 16 Filesystem created: Thu Jun 15 22:39:33 2017 Last mount time: Wed Jun 28 13:51:31 2017 Last write time: Wed Jun 28 13:51:31 2017 Mount count: 5 Maximum mount count: -1 Last checked: Thu Nov 19 19:00:00 2009 Check interval: 0 (<none>) Lifetime writes: 67 MB Reserved blocks uid: 0 (user root) Reserved blocks gid: 0 (group root) First inode: 11 Inode size: 256 Required extra isize: 32 Desired extra isize: 32 Journal inode: 8 Default directory hash: half_md4 Directory Hash Seed: 66aa0e7f-57da-41d0-86f7-d93270e53030 Journal backup: inode blocks
如何调整文件系统以获得更多的inode？
这是docker中overlay存储驱动程序的一个已知问题，由overlay2驱动程序解决。 新的cos-61 版本使用docker 17.03和overlay2存储驱动程序。 你可以试一下，看看问题是否再次发生？
谢谢！
编辑 ： 截至2017年10月10日，GKE正在使用Docker 17.03的COS 61镜像提供kubernetes 1.6.11和1.7.8： https ://cloud.google.com/container-engine/release-notes#october- 10-2017
我在GKE上使用COS 57.9202.64.0（docker 1.11.2 with overlay driver）也遇到同样的问题，并用spotify的docker-gc解决了这个问题：
# docker run --rm --userns host -v /var/run/docker.sock:/var/run/docker.sock -v /etc:/etc spotify/docker-gc
之前：
# df -hi /var/lib/docker/ Filesystem Inodes IUsed IFree IUse% Mounted on /dev/sda1 6.0M 5.0M 1.1M 83% /var # df -h /var/lib/docker/ Filesystem Size Used Avail Use% Mounted on /dev/sda1 95G 84G 11G 89% /var # du --inodes -s /var/lib/docker/* 180 /var/lib/docker/containers 4093 /var/lib/docker/image 4 /var/lib/docker/network 4906733 /var/lib/docker/overlay 1 /var/lib/docker/tmp 1 /var/lib/docker/trust 25 /var/lib/docker/volumes
后：
# df -hi /var/lib/docker/ Filesystem Inodes IUsed IFree IUse% Mounted on /dev/sda1 6.0M 327K 5.7M 6% /var/lib/docker # df -h /var/lib/docker/ Filesystem Size Used Avail Use% Mounted on /dev/sda1 95G 6.6G 88G 7% /var/lib/docker # du --inodes -s /var/lib/docker/* 218 /var/lib/docker/containers 1792 /var/lib/docker/image 4 /var/lib/docker/network 279002 /var/lib/docker/overlay 1 /var/lib/docker/tmp 1 /var/lib/docker/trust 25 /var/lib/docker/volumes
笔记：
使用通常的docker rmi $(docker images --filter "dangling=true" -q --no-trunc)和docker rm $(docker ps -qa --no-trunc --filter "status=exited")没有有助于恢复/var/lib/docker/overlay 。
另外，我不相信可以将GKE节点池升级到COS 61映像。