| kata-irc-bot | <sebastian.hasler> Hi! One of our Kata containers on our bare metal Kubernetes cluster doesn't start. containerd logs the following related log: ```May 17 12:58:56 kube02 kata[11781]: time="2022-05-17T12:58:56.804009548+02:00" level=error msg="createContainer failed" error="rpc error: code = Internal desc = failed to mount \"/run/kata-containers/shared/containers/75d60680125b6e552558afa32f16a87b428df50d5fc1e7dcfc62aa0d33a6424e/rootfs\" to | 11:02 |
|---|---|---|
| kata-irc-bot | \"/run/kata-containers/75d60680125b6e552558afa32f16a87b428df50d5fc1e7dcfc62aa0d33a6424e/rootfs\", with error: EMFILE: Too many open files" name=containerd-shim-v2 pid=11781 sandbox=0353aa2396a18862524d991584e09cebb85129c4f603ad83757029c8746ec349 source=virtcontainers subsystem=kata_agent May 17 12:58:56 kube02 containerd[4545]: time="2022-05-17T12:58:56.804009548+02:00" level=error msg="createContainer failed" error="rpc error: code = Internal | 11:02 |
| kata-irc-bot | desc = failed to mount \"/run/kata-containers/shared/containers/75d60680125b6e552558afa32f16a87b428df50d5fc1e7dcfc62aa0d33a6424e/rootfs\" to \"/run/kata-containers/75d60680125b6e552558afa32f16a87b428df50d5fc1e7dcfc62aa0d33a6424e/rootfs\", with error: EMFILE: Too many open files" name=containerd-shim-v2 pid=11781 sandbox=0353aa2396a18862524d991584e09cebb85129c4f603ad83757029c8746ec349 source=virtcontainers subsystem=kata_agent May 17 12:58:57 | 11:02 |
| kata-irc-bot | kube02 kata[11781]: time="2022-05-17T12:58:57.316292453+02:00" level=error msg="container create failed" container=75d60680125b6e552558afa32f16a87b428df50d5fc1e7dcfc62aa0d33a6424e error="rpc error: code = Internal desc = failed to mount \"/run/kata-containers/shared/containers/75d60680125b6e552558afa32f16a87b428df50d5fc1e7dcfc62aa0d33a6424e/rootfs\" to | 11:02 |
| kata-irc-bot | \"/run/kata-containers/75d60680125b6e552558afa32f16a87b428df50d5fc1e7dcfc62aa0d33a6424e/rootfs\", with error: EMFILE: Too many open files" name=containerd-shim-v2 pid=11781 sandbox=0353aa2396a18862524d991584e09cebb85129c4f603ad83757029c8746ec349 source=virtcontainers subsystem=container May 17 12:58:57 kube02 containerd[4545]: time="2022-05-17T12:58:57.316292453+02:00" level=error msg="container create failed" | 11:02 |
| kata-irc-bot | container=75d60680125b6e552558afa32f16a87b428df50d5fc1e7dcfc62aa0d33a6424e error="rpc error: code = Internal desc = failed to mount \"/run/kata-containers/shared/containers/75d60680125b6e552558afa32f16a87b428df50d5fc1e7dcfc62aa0d33a6424e/rootfs\" to \"/run/kata-containers/75d60680125b6e552558afa32f16a87b428df50d5fc1e7dcfc62aa0d33a6424e/rootfs\", with error: EMFILE: Too many open files" name=containerd-shim-v2 pid=11781 | 11:02 |
| kata-irc-bot | sandbox=0353aa2396a18862524d991584e09cebb85129c4f603ad83757029c8746ec349 source=virtcontainers subsystem=container May 17 12:58:57 kube02 containerd[4545]: time="2022-05-17T12:58:57.316452706+02:00" level=warning msg="Could not umount" container=75d60680125b6e552558afa32f16a87b428df50d5fc1e7dcfc62aa0d33a6424e error="no such file or directory" | 11:02 |
| kata-irc-bot | host-path=/run/kata-containers/shared/sandboxes/0353aa2396a18862524d991584e09cebb85129c4f603ad83757029c8746ec349/mounts/75d60680125b6e552558afa32f16a87b428df50d5fc1e7dcfc62aa0d33a6424e-d5a6d6afdb587735-mailman-web-data name=containerd-shim-v2 pid=11781 sandbox=0353aa2396a18862524d991584e09cebb85129c4f603ad83757029c8746ec349 source=virtcontainers subsystem=container May 17 12:58:57 kube02 containerd[4545]: | 11:02 |
| kata-irc-bot | time="2022-05-17T12:58:57.316509477+02:00" level=error msg="rollback failed unmountHostMounts()" container=75d60680125b6e552558afa32f16a87b428df50d5fc1e7dcfc62aa0d33a6424e error="no such file or directory" name=containerd-shim-v2 pid=11781 sandbox=0353aa2396a18862524d991584e09cebb85129c4f603ad83757029c8746ec349 source=virtcontainers subsystem=container May 17 12:58:57 kube02 containerd[4545]: time="2022-05-17T12:58:57.316575101+02:00" | 11:02 |
| kata-irc-bot | level=warning error="no such file or directory" name=containerd-shim-v2 pid=11781 sandbox=0353aa2396a18862524d991584e09cebb85129c4f603ad83757029c8746ec349 share-dir=/run/kata-containers/shared/sandboxes/0353aa2396a18862524d991584e09cebb85129c4f603ad83757029c8746ec349/mounts/75d60680125b6e552558afa32f16a87b428df50d5fc1e7dcfc62aa0d33a6424e/rootfs source=virtcontainers subsystem=mount May 17 12:58:57 kube02 kata[11781]: | 11:02 |
| kata-irc-bot | time="2022-05-17T12:58:57.316452706+02:00" level=warning msg="Could not umount" container=75d60680125b6e552558afa32f16a87b428df50d5fc1e7dcfc62aa0d33a6424e error="no such file or directory" host-path=/run/kata-containers/shared/sandboxes/0353aa2396a18862524d991584e09cebb85129c4f603ad83757029c8746ec349/mounts/75d60680125b6e552558afa32f16a87b428df50d5fc1e7dcfc62aa0d33a6424e-d5a6d6afdb587735-mailman-web-data name=containerd-shim-v2 pid=11781 | 11:02 |
| kata-irc-bot | sandbox=0353aa2396a18862524d991584e09cebb85129c4f603ad83757029c8746ec349 source=virtcontainers subsystem=container May 17 12:58:57 kube02 containerd[4545]: time="2022-05-17T12:58:57.316651838+02:00" level=warning msg="Could not remove container share dir" error="no such file or directory" name=containerd-shim-v2 pid=11781 sandbox=0353aa2396a18862524d991584e09cebb85129c4f603ad83757029c8746ec349 | 11:02 |
| kata-irc-bot | share-dir=/run/kata-containers/shared/sandboxes/0353aa2396a18862524d991584e09cebb85129c4f603ad83757029c8746ec349/mounts/75d60680125b6e552558afa32f16a87b428df50d5fc1e7dcfc62aa0d33a6424e source=virtcontainers subsystem="filesystem share" May 17 12:58:57 kube02 kata[11781]: time="2022-05-17T12:58:57.316509477+02:00" level=error msg="rollback failed unmountHostMounts()" container=75d60680125b6e552558afa32f16a87b428df50d5fc1e7dcfc62aa0d33a6424e | 11:02 |
| kata-irc-bot | error="no such file or directory" name=containerd-shim-v2 pid=11781 sandbox=0353aa2396a18862524d991584e09cebb85129c4f603ad83757029c8746ec349 source=virtcontainers subsystem=container May 17 12:58:57 kube02 kata[11781]: time="2022-05-17T12:58:57.316575101+02:00" level=warning error="no such file or directory" name=containerd-shim-v2 pid=11781 sandbox=0353aa2396a18862524d991584e09cebb85129c4f603ad83757029c8746ec349 | 11:02 |
| kata-irc-bot | share-dir=/run/kata-containers/shared/sandboxes/0353aa2396a18862524d991584e09cebb85129c4f603ad83757029c8746ec349/mounts/75d60680125b6e552558afa32f16a87b428df50d5fc1e7dcfc62aa0d33a6424e/rootfs source=virtcontainers subsystem=mount May 17 12:58:57 kube02 kata[11781]: time="2022-05-17T12:58:57.316651838+02:00" level=warning msg="Could not remove container share dir" error="no such file or directory" name=containerd-shim-v2 pid=11781 | 11:02 |
| kata-irc-bot | sandbox=0353aa2396a18862524d991584e09cebb85129c4f603ad83757029c8746ec349 share-dir=/run/kata-containers/shared/sandboxes/0353aa2396a18862524d991584e09cebb85129c4f603ad83757029c8746ec349/mounts/75d60680125b6e552558afa32f16a87b428df50d5fc1e7dcfc62aa0d33a6424e source=virtcontainers subsystem="filesystem share" May 17 12:58:58 kube02 containerd[4545]: time="2022-05-17T12:58:58.061802576+02:00" level=warning msg="cleaning up after shim | 11:02 |
| kata-irc-bot | disconnected" id=75d60680125b6e552558afa32f16a87b428df50d5fc1e7dcfc62aa0d33a6424e namespace=k8s.io May 17 12:58:58 kube02 containerd[4545]: time="2022-05-17T12:58:58.135845707+02:00" level=error msg="copy shim log" error="read /proc/self/fd/2766: file already closed" May 17 12:58:58 kube02 containerd[4545]: time="2022-05-17T12:58:58.136311651+02:00" level=error msg="Failed to pipe stdout of container | 11:02 |
| kata-irc-bot | \"75d60680125b6e552558afa32f16a87b428df50d5fc1e7dcfc62aa0d33a6424e\"" error="reading from a closed fifo" May 17 12:58:58 kube02 containerd[4545]: time="2022-05-17T12:58:58.136551103+02:00" level=error msg="Failed to pipe stderr of container \"75d60680125b6e552558afa32f16a87b428df50d5fc1e7dcfc62aa0d33a6424e\"" error="reading from a closed fifo" May 17 12:58:58 kube02 containerd[4545]: time="2022-05-17T12:58:58.140932815+02:00" level=error | 11:02 |
| kata-irc-bot | msg="StartContainer for \"75d60680125b6e552558afa32f16a87b428df50d5fc1e7dcfc62aa0d33a6424e\" failed" error="failed to create containerd task: failed to create shim: failed to mount \"/run/kata-containers/shared/containers/75d60680125b6e552558afa32f16a87b428df50d5fc1e7dcfc62aa0d33a6424e/rootfs\" to \"/run/kata-containers/75d60680125b6e552558afa32f16a87b428df50d5fc1e7dcfc62aa0d33a6424e/rootfs\", with error: EMFILE: Too many open files: | 11:02 |
| kata-irc-bot | unknown"``` Any ideas? The containerd systemd service on the node has `LimitNOFILE=infinity` . The node has sysctl values `fs.file-max = 9223372036854775807` and `fs.nr_open = 52706963` . According to `lsof | wc -l` there are around 8 million open files on the node. | 11:02 |
| kata-irc-bot | <liuj97> it's a known issue, I think,:) | 17:58 |
| kata-irc-bot | <liuj97> It may be solved by using nydus image format and some sort of overlayfs in guest. | 17:59 |
| kata-irc-bot | <liuj97> that is to avoid using virtiofs passthrough mode | 18:00 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!