Friday, 2022-01-14

« Thursday, 2022-01-13 Index Wednesday, 2022-01-19 »
kata-irc-bot<fighterhitx> Thanks @julio.montes,  I replaced `configuration.toml` with `configuration-clh.toml` , and then copied `cloud-hypervisor` and `virtiofsd` in `kata-static-1.12.1-x86_64.tar.xz` to `/usr/bin/` directory, but when creating a container by  `docker run  -it  --runtime=kata-runtime  --device /dev/vfio/84  --cap-add=ALL --cpus 10  debian:stretch /bin/bash`, the following error will be reported: `docker: Error response from daemon: OCI02:14
kata-irc-botruntime create failed: error: 500 reason: VmBoot(VmBoot(DeviceManager(CreateVirtioFs(VhostUserGetFeatures(VhostUserProtocol(SocketBroken(Os { code: 104, kind: ConnectionReset, message: "Connection reset by peer" } )))))))): unknown.` `ERRO[0001] error waiting for container: context canceled` ,02:14
kata-irc-bot<fighterhitx> I only change the `kernel` in the configuration.toml(override by `configuration-clh.toml`) to GPU supported kernel which built according to the  https://github.com/kata-containers/kata-containers/blob/main/docs/use-cases/Nvidia-GPU-passthrough-and-Kata.md, and container can be created normally when using qemu.02:25
kata-irc-bot<samuel.ortiz> @fighterhitx cloud hypervisor 0.11 is a fairly old version. I suggest you use a more recent version, v20.2 is that one that’s officially supported upstream09:23
kata-irc-bot<fighterhitx> Hi @samuel.ortiz, when I use v20.2.0 cloud hypervisor, it reports following error: ```root@ai-2080ti-27:/usr/share/defaults/kata-containers# cloud-hypervisor -V cloud-hypervisor v20.2.0 root@ai-2080ti-27:/usr/share/defaults/kata-containers# docker run  -it  --runtime=kata-runtime  --device /dev/vfio/85  --cap-add=ALL --cpus 10  debian:stretch /bin/bash docker: Error response from daemon: OCI runtime create failed: error: 500 09:51
kata-irc-botreason: VmBoot(VmBoot(DeviceManager(CreateVirtioFs(VhostUserGetFeatures(VhostUserProtocol(SocketBroken(Os { code: 104, kind: ConnectionReset, message: "Connection reset by peer" }))))))): unknown. ERRO[0001] error waiting for container: context canceled ```09:51
kata-irc-bot<julio.montes> @fighterhitx according to the versions.yaml cloud hypervisor 0.11 is supported in that release https://github.com/kata-containers/runtime/blob/1.12.1/versions.yaml#L73-L7813:40
kata-irc-bot<julio.montes> remove `--device /dev/vfio/84  --cap-add=ALL`  from the cmdline, maybe that's the error13:42
kata-irc-bot<fighterhitx> @julio.montes unluckily it still return error: ```root@ai-2080ti-27:~# kata-runtime -v kata-runtime  : 1.12.1    commit   : <<unknown>>    OCI specs: 1.0.1-dev root@ai-2080ti-27:~# cloud-hypervisor -V cloud-hypervisor v0.11.0 root@ai-2080ti-27:~# docker run  -it  --runtime=kata-runtime    debian:stretch /bin/bash docker: Error response from daemon: OCI runtime create failed: error: 500  reason:14:28
kata-irc-botVmBoot(VmBoot(DeviceManager(CreateVirtioFs(VhostUserGetFeatures(VhostUserProtocol(SocketBroken(Os { code: 104, kind: ConnectionReset, message: "Connection reset by peer" }))))))): unknown. ERRO[0001] error waiting for container: context canceled```14:28
kata-irc-bot<julio.montes> @fighterhitx it works for me, I'm using this tarball https://github.com/kata-containers/runtime/releases/tag/1.12.114:36
kata-irc-bot<julio.montes> $ docker run -ti --runtime kata mirror.gcr.io/library/ubuntu bash root@052759fca03c:/# dmesg | grep -i cloud [    0.000000] DMI: Cloud Hypervisor cloud-hypervisor, BIOS 0 [    0.001159] ACPI: RSDP 0x00000000000A0000 000024 (v02 CLOUDH)14:37
kata-irc-bot<fighterhitx> May this be related to the version of docker or containerd? On my host, their versions are as follows: ```root@ai-2080ti-27:~# docker version Client: Docker Engine - Community  Version:           19.03.12  API version:       1.40  Go version:        go1.13.10  Git commit:        48a66213fe  Built:             Mon Jun 22 15:45:52 2020  OS/Arch:           linux/amd64  Experimental:      false  Server: Docker Engine - Community 14:47
kata-irc-botEngine:   Version:          19.03.12   API version:      1.40 (minimum version 1.12)   Go version:       go1.13.10   Git commit:       48a66213fe   Built:            Mon Jun 22 15:44:23 2020   OS/Arch:          linux/amd64   Experimental:     false  containerd:   Version:          1.2.13   GitCommit:        7ad184331fa3e55e52b890ea95e65ba581ae3429  nvidia:   Version:          1.0.0-rc10   GitCommit:       14:47
kata-irc-botdc9208a3303feef5b3839f4323d9beb36df0a9dd-dirty  docker-init:   Version:          0.18.0   GitCommit:        fec3683  root@ai-2080ti-27:~# ctr version Client:   Version:  1.2.13   Revision: 7ad184331fa3e55e52b890ea95e65ba581ae3429  Server:   Version:  1.2.13   Revision: 7ad184331fa3e55e52b890ea95e65ba581ae3429``` And, I got some errors from `journalctl -u docker -f` when I create the container using the same command: ```Jan 14 22:47:1514:47
kata-irc-botai-2080ti-27 dockerd[2331]: time="2022-01-14T22:47:15.487386388+08:00" level=error msg="stream copy error: reading from a closed fifo" Jan 14 22:47:15 ai-2080ti-27 dockerd[2331]: time="2022-01-14T22:47:15.741376407+08:00" level=error msg="6766e5e172a3c926d723675da08f8fc196fd0acfea626c489aac174229b8b81b cleanup: failed to delete container from containerd: no such container" Jan 14 22:47:15 ai-2080ti-27 dockerd[2331]:14:47
kata-irc-bottime="2022-01-14T22:47:15.741460230+08:00" level=error msg="Handler for POST /v1.40/containers/6766e5e172a3c926d723675da08f8fc196fd0acfea626c489aac174229b8b81b/start returned error: OCI runtime create failed: error: 500  reason: VmBoot(VmBoot(DeviceManager(CreateVirtioFs(VhostUserGetFeatures(VhostUserProtocol(SocketBroken(Os { code: 104, kind: ConnectionReset, message: \"Connection reset by peer\" }))))))): unknown"```14:47
kata-irc-bot<julio.montes> I don't think so15:07
kata-irc-bot<julio.montes> maybe reinstalling kata fix the issue15:08
kata-irc-bot<julio.montes> I'm not installing it from deb packages, I'm using the tarball from release page15:09
kata-irc-bot<fighterhitx> @julio.montes How to install according to tarball? Copy the files in the opt/kata/bin directory of tarball  to /usr/bin directory of my host?15:15
kata-irc-bot<fighterhitx> I installed Kata1.12 according to this debian-installation-guide  before.15:18
kata-irc-bot<julio.montes> @fighterhitx $ sudo tar -xf $TARBALL -C /15:28
kata-irc-bot<julio.montes> replace $TARBALL with tarball filename15:28
kata-irc-bot<julio.montes> sudo rm -rf /usr/bin/kata-runtime15:29
kata-irc-bot<julio.montes> sudo ln -sf /opt/kata/bin/kata-runtime /usr/bin/kata-runtime15:29
kata-irc-bot<fighterhitx> @julio.montes Really appreciate for your kindly help, I'll try it right away!15:30
kata-irc-bot<julio.montes> sudo rm -f /opt/kata/share/defaults/kata-containers/configuration.toml15:30
kata-irc-bot<julio.montes> sudo ln -sf /opt/kata/share/defaults/kata-containers/configuration-clh.toml /opt/kata/share/defaults/kata-containers/configuration.toml15:31
kata-irc-bot<julio.montes> @fighterhitx yw15:31
kata-irc-bot<fighterhitx> sorry @julio.montes, but still report the same error without luck ```root@ai-2080ti-27:/opt/kata/share/defaults/kata-containers# docker run  -it  --runtime=kata-runtime     debian:stretch /bin/bash docker: Error response from daemon: OCI runtime create failed: error: 500  reason: VmBoot(VmBoot(DeviceManager(CreateVirtioFs(VhostUserGetFeatures(VhostUserProtocol(SocketBroken(Os { code: 104, kind: ConnectionReset, message:16:03
kata-irc-bot"Connection reset by peer" }))))))): unknown. ERRO[0001] error waiting for container: context canceled``` 16:03
kata-irc-bot<julio.montes> @fighterhitx is vhost-vsock module loaded?16:05
kata-irc-bot<julio.montes> $ ls -l /dev/vhost-vsock16:05
kata-irc-bot<julio.montes> lsmod | gre[ vsock16:05
kata-irc-bot<julio.montes> $ lsmod | grep vsock16:05
kata-irc-bot<fighterhitx> seems not ```root@ai-2080ti-27:~# ls -l /dev/vhost-vsock ls: cannot access '/dev/vhost-vsock': No such file or directory  root@ai-2080ti-27:~# lsmod | grep vsock root@ai-2080ti-27:~# ```16:06
kata-irc-bot<julio.montes> ahh, that's the problem16:08
kata-irc-bot<fighterhitx> Yes, now have ```root@ai-2080ti-27:~# lsmod | grep vsock vhost_vsock            20480  0 vmw_vsock_virtio_transport_common    28672  1 vhost_vsock vsock                  36864  2 vhost_vsock,vmw_vsock_virtio_transport_common vhost                  45056  2 vhost_vsock,vhost_net root@ai-2080ti-27:~# ls -l /dev/vhost-vsock crw------- 1 root root 10, 57 Jan 15 00:08 /dev/vhost-vsock```16:10
kata-irc-bot<fighterhitx> but..16:10
kata-irc-bot<fighterhitx> ```root@ai-2080ti-27:~# docker run  -it  --runtime=kata-runtime     debian:stretch /bin/bash docker: Error response from daemon: OCI runtime create failed: error: 500  reason: VmBoot(VmBoot(DeviceManager(CreateVirtioFs(VhostUserGetFeatures(VhostUserProtocol(SocketBroken(Os { code: 104, kind: ConnectionReset, message: "Connection reset by peer" }))))))): unknown. ERRO[0001] error waiting for container: context canceled ```16:10
kata-irc-bot<julio.montes> is kvm loaded?16:16
kata-irc-bot<fighterhitx> @julio.montes When I use qemu, it's normal.16:17
kata-irc-bot<julio.montes> can you check fuse module?16:23
kata-irc-bot<julio.montes> $ lsmod | grep fuse16:23
kata-irc-bot<fighterhitx> yep16:23
kata-irc-bot<fighterhitx> root@ai-2080ti-27:~# lsmod | grep fuse root@ai-2080ti-27:~#16:23
kata-irc-bot<fighterhitx> ```modprobe fuse?```16:24
kata-irc-bot<julio.montes> yes16:24
kata-irc-bot<fighterhitx> ```root@ai-2080ti-27:~# modprobe fuse root@ai-2080ti-27:~# docker run  -it  --runtime=kata-runtime     debian:stretch /bin/bash docker: Error response from daemon: OCI runtime create failed: error: 500  reason: VmBoot(VmBoot(DeviceManager(CreateVirtioFs(VhostUserGetFeatures(VhostUserProtocol(SocketBroken(Os { code: 104, kind: ConnectionReset, message: "Connection reset by peer" }))))))): unknown. ERRO[0001] error waiting for16:24
kata-irc-botcontainer: context canceled  root@ai-2080ti-27:~# lsmod |grep fuse fuse                   98304  1```16:24
kata-irc-bot<julio.montes> ouch!16:26
kata-irc-bot<julio.montes> can you enable kata debug? and run it again16:27
kata-irc-bot<julio.montes> journalctl -b- t kata-runtime16:27
kata-irc-bot<julio.montes> sudo ^16:27
kata-irc-bot<fighterhitx> Yes, is `journalctl -b- t kata-runtime`  enough? Sorry  I haven't debugged kata yet16:29
kata-irc-bot<julio.montes> yes16:31
kata-irc-bot<julio.montes> `sudo journalctl -b- t kata-runtime`16:32
kata-irc-bot<fighterhitx> ```root@ai-2080ti-27:~# grep enable_debug /opt/kata/share/defaults/kata-containers/configuration.toml  enable_debug = true enable_debug = true enable_debug = true enable_debug = true enable_debug = true enable_debug = true root@ai-2080ti-27:~# docker run  -it  --runtime=kata-runtime     debian:stretch /bin/bash docker: Error response from daemon: OCI runtime create failed: error: 500  reason:16:43
kata-irc-botVmBoot(VmBoot(DeviceManager(CreateVirtioFs(VhostUserGetFeatures(VhostUserProtocol(SocketBroken(Os { code: 104, kind: ConnectionReset, message: "Connection reset by peer" }))))))): unknown. ERRO[0000] error waiting for container: context canceled ``` 16:43
kata-irc-bot<fighterhitx> `journalctl -b -t kata-runtime`  output:16:47
kata-irc-bot<julio.montes> sorry, I'm not able to identify the issue, may be you are running out of memory  :thinking_face:17:07
kata-irc-bot<julio.montes> can you change `default_memory = 1024`17:07
kata-irc-bot<julio.montes> ?17:07
kata-irc-bot<fighterhitx> yes ican17:08
kata-irc-bot<julio.montes> maybe `kernel_params = "init=/usr/bin/kata-agent"`  can also help17:09
kata-irc-bot<fighterhitx> ok17:09
kata-irc-bot<fighterhitx> i'll try17:09
kata-irc-bot<fighterhitx> ```root@ai-2080ti-27:~# grep "default_memory\|kernel_params" /opt/kata/share/defaults/kata-containers/configuration.toml # For example, use `kernel_params = "vsyscall=emulate"` if you are having kernel_params = "init=/usr/bin/kata-agent" #default_memory = 2048 default_memory = 4096 root@ai-2080ti-27:~# docker run  -it  --runtime=kata-runtime     debian:stretch /bin/bash docker: Error response from daemon: OCI runtime create17:12
kata-irc-botfailed: error: 500  reason: VmBoot(VmBoot(DeviceManager(CreateVirtioFs(VhostUserGetFeatures(VhostUserProtocol(SocketBroken(Os { code: 104, kind: ConnectionReset, message: "Connection reset by peer" }))))))): unknown. ERRO[0001] error waiting for container: context canceled```17:13
kata-irc-bot<fighterhitx> 17:15
kata-irc-bot<julio.montes> no idea what's failing :(17:20
kata-irc-bot<fighterhitx> maybe I need uninstall the old kata using `apt` and reinstall17:23
« Thursday, 2022-01-13 Index Wednesday, 2022-01-19 »

Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!