| kata-irc-bot | <tim.rice> hello! we're experimenting with kata-containers 2.2 installed via snap on ubuntu 20.04 with CRI-O... have gotten containers to run but the crio log gets spammed a bit with error messages regarding events, for example ```time="2021-09-02T08:28:21.376817422Z" level=error msg="post event" error="failed to connect: dial unix: missing address" name=containerd-shim-v2 pid=38173 | 08:29 |
|---|---|---|
| kata-irc-bot | sandbox=eecec82e8d7f72ce41a6aad5586bf58c7bd75f04b226b72a37cfde20b7dd7b6d source=containerd-kata-shim-v2``` | 08:29 |
| kata-irc-bot | <tim.rice> can anyone point out which unix socket address its looking for and where to configure this? :slightly_smiling_face: | 08:30 |
| kata-irc-bot | <tim.rice> great project btw. | 08:31 |
| kata-irc-bot | <tim.rice> despite the errors everything seems to work | 08:31 |
| kata-irc-bot | <fidencio> What's the CRI-O version? | 10:01 |
| kata-irc-bot | <fidencio> Is that the only error you see in the journal? How did you configure kata-containers? | 10:01 |
| kata-irc-bot | <tim.rice> ```[crio.runtime.runtimes.kata] runtime_path = "/usr/local/bin/containerd-shim-kata-v2" runtime_type = "vm" runtime_root = "/run/kata-containers"``` is our relevant crio conf for kata. the documentation is a little contradictory in places but that is what we ended up with that got us up and running. | 10:12 |
| kata-irc-bot | <tim.rice> thats basically the only errors we see, crio then repeats them, i assume its just repeating the output it got from kata | 10:13 |
| kata-irc-bot | <tim.rice> and we're running 1.21.2 cri-o | 10:13 |
| kata-irc-bot | <tim.rice> actually thats not entirely true, heres all the error messages from the crio journal during a lifetime of a kata container ```Sep 02 08:50:54 pumped-gator kata[74004]: time="2021-09-02T08:50:54.189487508Z" level=error msg="evicting /tasks/start from queue because of retry count" Sep 02 08:50:54 pumped-gator kata[74004]: time="2021-09-02T08:50:54.189597329Z" level=error msg="evicting /tasks/create from queue because of retry count" | 10:14 |
| kata-irc-bot | Sep 02 08:50:54 pumped-gator crio[36585]: time="2021-09-02T08:50:54.189487508Z" level=error msg="evicting /tasks/start from queue because of retry count" Sep 02 08:50:54 pumped-gator crio[36585]: time="2021-09-02T08:50:54.189597329Z" level=error msg="evicting /tasks/create from queue because of retry count" Sep 02 08:50:55 pumped-gator kata[74004]: time="2021-09-02T08:50:55.761928735Z" level=error msg="forward event" error="failed to connect: | 10:14 |
| kata-irc-bot | dial unix: missing address" Sep 02 08:50:55 pumped-gator crio[36585]: time="2021-09-02T08:50:55.761928735Z" level=error msg="forward event" error="failed to connect: dial unix: missing address" Sep 02 08:50:55 pumped-gator kata[74004]: time="2021-09-02T08:50:55.769293886Z" level=error msg="forward event" error="failed to connect: dial unix: missing address" Sep 02 08:50:55 pumped-gator crio[36585]: time="2021-09-02T08:50:55.769293886Z" | 10:14 |
| kata-irc-bot | level=error msg="forward event" error="failed to connect: dial unix: missing address" Sep 02 08:50:58 pumped-gator kata[74004]: time="2021-09-02T08:50:58.636348333Z" level=error msg="evicting /tasks/create from queue because of retry count" Sep 02 08:50:58 pumped-gator crio[36585]: time="2021-09-02T08:50:58.636348333Z" level=error msg="evicting /tasks/create from queue because of retry count" Sep 02 08:50:58 pumped-gator kata[74004]: | 10:14 |
| kata-irc-bot | time="2021-09-02T08:50:58.638619791Z" level=error msg="evicting /tasks/start from queue because of retry count" Sep 02 08:50:58 pumped-gator crio[36585]: time="2021-09-02T08:50:58.638619791Z" level=error msg="evicting /tasks/start from queue because of retry count" Sep 02 08:51:01 pumped-gator kata[74004]: time="2021-09-02T08:51:01.763191166Z" level=error msg="evicting /tasks/exec-added from queue because of retry count" Sep 02 08:51:01 | 10:14 |
| kata-irc-bot | pumped-gator crio[36585]: time="2021-09-02T08:51:01.763191166Z" level=error msg="evicting /tasks/exec-added from queue because of retry count" Sep 02 08:51:01 pumped-gator kata[74004]: time="2021-09-02T08:51:01.770507211Z" level=error msg="evicting /tasks/exec-started from queue because of retry count" Sep 02 08:51:01 pumped-gator crio[36585]: time="2021-09-02T08:51:01.770507211Z" level=error msg="evicting /tasks/exec-started from queue | 10:14 |
| kata-irc-bot | because of retry count" Sep 02 09:04:24 pumped-gator kata[74004]: time="2021-09-02T09:04:24.916967709Z" level=error msg="post event" error="failed to connect: dial unix: missing address" name=containerd-shim-v2 pid=74004 sandbox=e5d7658f8c2dba8476d30c47e581a1133395d6d72d87359e440563b3a5841e74 source=containerd-kata-shim-v2 Sep 02 09:04:24 pumped-gator crio[36585]: time="2021-09-02T09:04:24.916967709Z" level=error msg="post event" error="failed | 10:14 |
| kata-irc-bot | to connect: dial unix: missing address" name=containerd-shim-v2 pid=74004 sandbox=e5d7658f8c2dba8476d30c47e581a1133395d6d72d87359e440563b3a5841e74 source=containerd-kata-shim-v2 Sep 02 09:04:25 pumped-gator kata[74004]: time="2021-09-02T09:04:25.917799461Z" level=error msg="forward event" error="failed to connect: dial unix: missing address" Sep 02 09:04:25 pumped-gator crio[36585]: time="2021-09-02T09:04:25.917799461Z" level=error | 10:14 |
| kata-irc-bot | msg="forward event" error="failed to connect: dial unix: missing address" Sep 02 09:04:27 pumped-gator kata[74004]: time="2021-09-02T09:04:27.918545649Z" level=error msg="forward event" error="failed to connect: dial unix: missing address" Sep 02 09:04:27 pumped-gator crio[36585]: time="2021-09-02T09:04:27.918545649Z" level=error msg="forward event" error="failed to connect: dial unix: missing address" Sep 02 09:04:30 pumped-gator kata[74004]: | 10:14 |
| kata-irc-bot | time="2021-09-02T09:04:30.91984731Z" level=error msg="forward event" error="failed to connect: dial unix: missing address" Sep 02 09:04:30 pumped-gator crio[36585]: time="2021-09-02T09:04:30.919847310Z" level=error msg="forward event" error="failed to connect: dial unix: missing address" Sep 02 09:04:34 pumped-gator kata[74004]: time="2021-09-02T09:04:34.920801487Z" level=error msg="forward event" error="failed to connect: dial unix: missing | 10:14 |
| kata-irc-bot | address" Sep 02 09:04:34 pumped-gator crio[36585]: time="2021-09-02T09:04:34.920801487Z" level=error msg="forward event" error="failed to connect: dial unix: missing address" Sep 02 09:04:39 pumped-gator kata[74004]: time="2021-09-02T09:04:39.921854766Z" level=error msg="forward event" error="failed to connect: dial unix: missing address" Sep 02 09:04:39 pumped-gator crio[36585]: time="2021-09-02T09:04:39.921854766Z" level=error msg="forward | 10:14 |
| kata-irc-bot | event" error="failed to connect: dial unix: missing address" Sep 02 09:04:45 pumped-gator kata[74004]: time="2021-09-02T09:04:45.922125434Z" level=error msg="evicting /tasks/exit from queue because of retry count" Sep 02 09:04:45 pumped-gator crio[36585]: time="2021-09-02T09:04:45.922125434Z" level=error msg="evicting /tasks/exit from queue because of retry count"``` | 10:14 |
| kata-irc-bot | <fidencio> Interesting, you CRI-O config is not wrong | 10:16 |
| kata-irc-bot | <fidencio> Let me past what we use with kata-deploy | 10:16 |
| kata-irc-bot | <fidencio> https://github.com/kata-containers/kata-containers/blob/716ab32acff938a972ab704c00[…]885ca8718ca0/tools/packaging/kata-deploy/scripts/kata-deploy.sh | 10:17 |
| kata-irc-bot | <fidencio> Out of curiosity, have you checked https://github.com/kata-containers/kata-containers/tree/main/snap ? | 10:18 |
| kata-irc-bot | <fgiudici> Wondering if the runtime_root configuration may be the cause of the failure. Guess we may need /run/vc there, as it is where we expect to find the socket: https://github.com/kata-containers/kata-containers/blob/stable-2.2/src/runtime/containerd-shim-v2/shim_management.go#L189 | 10:33 |
| kata-irc-bot | <fidencio> Good point, Francesco! | 10:34 |
| kata-irc-bot | <escita> Hello there, is there anyone know how to run kata-firecracker container with `ctr` client without getting k8s involved. I have `containerd` installed, and extracted all the tar ball of `kata 2.2.0` at path `/opt/` I could create and run and container using the following commands: `sudo ctr c create docker.io/library/ubuntu:latest ubuntu` `sudo ctr t start -d ubuntu` How do I run a `kata container with firecracker` using `ctr | 12:15 |
| kata-irc-bot | client` since `docker cli` does not support shimv2 architecture. :grinning: | 12:15 |
| kata-irc-bot | <jakob.naucke> `--runtime io.containerd.kata.v2`, the file `containerd-shim-kata-v2` must be in your PATH (you can create a link to the file in `/opt` in `/usr/local/bin` if you haven't already) | 13:03 |
| kata-irc-bot | <jakob.naucke> I don't usually do FC, but I think you have to copy (or edit?) the appropriate `configuration.toml` from the `/opt/kata` tree | 13:05 |
| kata-irc-bot | <escita> Hi Jakob, thanks for the info :slightly_smiling_face:. Yeah I could launch a container with default runtime by linking `containerd-shim-kata-v2` with `io.containerd.kata.v2` in `/usr/local/bin` , There is a `firecracker` binary under `/opt/kata/bin` , I haven't found the doc on how to launch a container with FC. | 13:22 |
| kata-irc-bot | <escita> Actually, I could only created the container by making symbolic link, but when I try to start it I get this error `ctr: runtime "io.containerd.kata.v2" binary not installed "containerd-shim-kata-v2": file does not exist: unknown` | 13:28 |
| kata-irc-bot | <anastassios.nanos> Hi there! in order to run kata with FC you'll first need to setup the devicemapper snapshotter then I guess you should make sure there's the relevant handler in /etc/containerd/config.toml eg ```[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.kata-fc] runtime_type = "io.containerd.kata-fc.v2"``` and then make sure there's a file in your PATH (eg `/usr/local/bin` named after the handler: ```# cat | 13:34 |
| kata-irc-bot | /usr/local/bin/containerd-shim-kata-fc-v2 #!/bin/bash KATA_CONF_FILE=/opt/kata/share/defaults/kata-containers/configuration-fc.toml /opt/kata/bin/containerd-shim-kata-v2 "$@"``` Finally run something like the following, after you've pulled the relevant container image (`ctr images pull --snapshotter devmapper docker.io/library/ubuntu:latest`): ```ctr run --snapshotter devmapper --runtime io.containerd.run.kata-fc.v2 -t --rm | 13:34 |
| kata-irc-bot | docker.io/library/ubuntu:latest ubuntu-kata-fc-test uname -a``` | 13:34 |
| kata-irc-bot | <anastassios.nanos> a rough post about getting kata to work with FC is the following: https://blog.cloudkernels.net/posts/kata-fc-k3s-k8s/ (full disclosure, to be polished ;)) | 14:07 |
| kata-irc-bot | <escita> Hi Ananos, thank you for the post, I will try it out tomorrow :slightly_smiling_face: and Yeah I managed to get `kata-qemu` container running. | 14:21 |
| kata-irc-bot | <shuo.chen> Hi team, we are testing the kata performance and find a wierd issue, we are using fio to test the virtio-scsi block device io performance. when are using this command in the guest vm: ```fio --randrepeat=0 --verify=0 --ioengine=libaio --direct=1 --gtod_reduce=1 --name=read_1bw --filename=/dev/xda --bs=256K --numjobs=5 --iodepth=64 --size=2G --readwrite=randwrite --time_based --ramp_time=2s --runtime=15s``` We find out that the IO | 23:17 |
| kata-irc-bot | is around 10GB/s (hit the cache on host side), But after remove the `direct=1` flag from fio, I expect a equal or higher IO performance, since guest page cache is involved but I only saw around 1GB/s. Anyone know the potential reason? | 23:17 |
| kata-irc-bot | <fidencio> cc @rlk | 23:23 |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!