Thursday, 2021-04-01

« Wednesday, 2021-03-31 Index Friday, 2021-04-02 »
kata-irc-bot2<raravena80> Woohooo :nerd_face: CTL03:24
*** egernst has joined #kata-dev03:34
*** egernst has quit IRC03:40
*** egernst has joined #kata-dev04:10
*** hbrueckner has joined #kata-dev06:49
*** hbrueckner has quit IRC06:57
*** ailan_ has joined #kata-dev07:02
*** fgiudici has joined #kata-dev07:05
*** dklyle has quit IRC07:05
*** jodh has joined #kata-dev07:16
*** sameo has joined #kata-dev07:29
*** davidgiluk has joined #kata-dev08:07
*** crobinso has joined #kata-dev12:59
*** dklyle has joined #kata-dev13:36
kata-irc-bot2<efoster> Ooh, I remember this. Are you running with the version of QEMU that has more memory slots?15:05
kata-irc-bot2<efoster> https://github.com/kata-containers/runtime/issues/279515:06
kata-irc-bot2<eric.ernst> Yeah, I had plenty of slots -- issue ended up being the amount of memory left available for those slots. All good now!16:16
kata-irc-bot2<eric.ernst> Journal chattiness when enabling info, debug :thread:16:17
kata-irc-bot2<eric.ernst> We have a few sources of chatter in journal: • trace span info16:17
kata-irc-bot2<fidencio> That's a good start16:20
kata-irc-bot2<fidencio> https://github.com/kata-containers/kata-containers/issues/160016:20
kata-irc-bot2<eric.ernst> See: •  https://github.com/kata-containers/kata-containers/pull/1616 • https://github.com/kata-containers/govmm/pull/16616:20
kata-irc-bot2<eric.ernst> Let me know what ya'll think!16:20
kata-irc-bot2<fidencio> I think the main problem is that, AFAIU, we don't have the concept of log_level on kata-contaibners16:21
kata-irc-bot2<fidencio> it's either on or off :slightly_smiling_face:16:21
kata-irc-bot2<fidencio> And that's annoying16:21
kata-irc-bot2<eric.ernst> In my setup (containerd), I can set it.16:21
kata-irc-bot2<fidencio> Well, you have the log_level on containerd side16:21
kata-irc-bot2<eric.ernst> Yeah, exactly. Since they are the consumer of our log output, yep, I have to filter there.16:21
kata-irc-bot2<eric.ernst> So, when doing debug, we do definitely have control over the verobsity of what we see in the output (ie, move from info to debug to warn)16:22
kata-irc-bot2<eric.ernst> isn't as nice as having just our own, but I think that's just part of the v2-shim pattern.16:23
kata-irc-bot2<fidencio> There are several problems here, and I'd appreciate if you can read my RFE.  Basically, containerd has the concept of log_level, kata-containers doesn't. containerd, when using "debug" will turn "debug" logs *on* for Kata Containers16:23
kata-irc-bot2<fidencio> But there's no way to actually tune kata-containers debug log_level (as there's no concept, there's never been a concept) of log_level on Kata Containers16:24
kata-irc-bot2<fidencio> it's always been on or off.16:24
kata-irc-bot2<fidencio> I'd appreciate some feedback on the RFE opened (https://github.com/kata-containers/kata-containers/issues/1600) as I'm super into implementing that sooner than later, and I'm happy to touch all the needed parts.16:24
kata-irc-bot2<fidencio> https://github.com/containerd/containerd/blob/9a9bd097564b0973bfdb0b39bf8262aa1b7da6aa/runtime/v2/binary.go#L61-L6316:29
kata-irc-bot2<fidencio> That looks ... quite weird, at least.16:30
kata-irc-bot2<fidencio> Early dinner time, I'll be back in 2hours or so. :slightly_smiling_face:16:31
kata-irc-bot2<eric.ernst> k.16:31
kata-irc-bot2<eric.ernst> yeah, i can ping contaiNerds about this.16:32
kata-irc-bot2<fidencio> I'd like to have an agreement on our side that we'd like to go with approach mentioned in the RFE and once we do that, we can sync, as the first thing, with containerd folks.16:33
kata-irc-bot2<fidencio> I'd expect the transition on their side would be the last bit pushed.16:33
kata-irc-bot2<fidencio> So they do that for our releases that are already patched and won't flood the logs16:33
kata-irc-bot2<eric.ernst> AFAIU, we'd want a seperate option at in the CRI runtime to enable logs for the shim, and for the register shim, have a configuration option for levels, right?16:36
kata-irc-bot2<eric.ernst> Is that the tl;dr?16:37
kata-irc-bot2<eric.ernst> Or -- rather -- to have an on/off at the CRI runtime level to consume our logs.16:37
kata-irc-bot2<eric.ernst> and we'd set the log level on our own side?16:38
fidencio@eric.ernrst, the latter16:38
kata-irc-bot2<eric.ernst> ack - let's talk after first dinner or tomorrow :)16:39
kata-irc-bot2<eric.ernst> I'll chat w/ the containerd guys for initial feedback (ask them to comment in your issue)16:39
kata-irc-bot2<eric.ernst> this sgtm.16:39
*** jodh has quit IRC17:06
*** fgiudici has quit IRC18:50
*** ailan_ has quit IRC19:19
*** davidgiluk has quit IRC19:31
*** crobinso has quit IRC20:20
*** amorenoz has quit IRC20:21
*** amorenoz has joined #kata-dev20:21
kata-irc-bot2<eric.ernst> Aaaaaah, was doing some debugging with 2.1.alpha0 and was running into spurious issues. Turns out @tim hit this problem already, and is fixed in 2.1-alpha120:45
kata-irc-bot2<eric.ernst> Nice work. Hopefully this is the last of the async agent fixes!20:46
*** sameo has quit IRC22:47
kata-irc-bot2<gmmaharaj> hello, anyone tested kata-deploy + k8s for 2.0.1? I am trying to test it out with a small VM and virtiofs always fails with the error below. ```Apr 01 22:59:58 u-01 containerd[17975]: time="2021-04-01T22:59:58.854137446Z" level=info msg="StartContainer for \"6162629ffcc61c5d8e31e83adc901d9bfde611bdc8c3cf6fd873c8313c995108\"" Apr 01 22:59:58 u-01 virtiofsd[19858]: [ID: 00000004] fv_queue_thread: Got queue event on Queue 123:08
kata-irc-bot2                                                                                                                                                                Apr 01 22:59:58 u-01 virtiofsd[19858]: [ID: 00000004] fv_queue_thread: Queue 1 gave evalue: 1 available: in: 144 out: 105                                                                                                                                                       Apr 01 22:59:5823:08
kata-irc-bot2u-01 virtiofsd[19858]: [ID: 00000004] fv_queue_thread: Waiting for Queue 1 event Apr 01 22:59:58 u-01 virtiofsd[19858]: [ID: 00000005] fv_queue_worker: elem 1: with 2 out desc of length 105                                                                                                                                                                    Apr 01 22:59:58 u-01 virtiofsd[19858]: [ID: 00000005] unique: 76, opcode: LOOKUP (1), nodeid: 1,23:08
kata-irc-bot2insize: 105, pid: 51 Apr 01 22:59:58 u-01 virtiofsd[19858]: [ID: 00000005] lo_lookup(parent=1, name=6162629ffcc61c5d8e31e83adc901d9bfde611bdc8c3cf6fd873c8313c995108)                                                                                                                                Apr 01 22:59:58 u-01 virtiofsd[19858]: [ID: 00000005]   1/6162629ffcc61c5d8e31e83adc901d9bfde611bdc8c3cf6fd873c8313c995108 -> 1323:08
kata-irc-bot2                                                                                                                  Apr 01 22:59:58 u-01 virtiofsd[19858]: [ID: 00000005]    unique: 76, success, outsize: 144                                                                                                                                                                                      Apr 01 22:59:58 u-01 virtiofsd[19858]: [ID: 00000005]23:08
kata-irc-bot2virtio_send_msg: elem 1: with 2 in desc of length 144 Apr 01 22:59:58 u-01 virtiofsd[19858]: [ID: 00000004] fv_queue_thread: Got queue event on Queue 1                                                                                                                                                                               Apr 01 22:59:58 u-01 virtiofsd[19858]: [ID: 00000004] fv_queue_thread: Queue 1 gave evalue: 1 available: in: 144 out: 47 Apr23:08
kata-irc-bot201 22:59:58 u-01 virtiofsd[19858]: [ID: 00000004] fv_queue_thread: Waiting for Queue 1 event                                                                                                                                                                                Apr 01 22:59:58 u-01 virtiofsd[19858]: [ID: 00000005] fv_queue_worker: elem 1: with 2 out desc of length 47 Apr 01 22:59:58 u-01 virtiofsd[19858]: [ID: 00000005] unique: 78, opcode:23:08
kata-irc-bot2LOOKUP (1), nodeid: 13, insize: 47, pid: 51 Apr 01 22:59:58 u-01 virtiofsd[19858]: [ID: 00000005] lo_lookup(parent=13, name=rootfs) Apr 01 22:59:58 u-01 virtiofsd[19858]: [ID: 00000005]   13/rootfs -> 14 Apr 01 22:59:58 u-01 virtiofsd[19858]: [ID: 00000005]    unique: 78, success, outsize: 144 Apr 01 22:59:58 u-01 virtiofsd[19858]: [ID: 00000005] virtio_send_msg: elem 1: with 2 in desc of length 144 Apr 01 22:59:58 u-01 kata[19838]:23:08
kata-irc-bot2time="2021-04-01T22:59:58.965535498Z" level=error msg="createContainer failed" error="rpc error: code = Internal desc = unable to write to a control group file" pid=19838 sandbox=8aee44b3d558471182048faa704d805fc7bce25df77c1328e3f672d58054fe27 source=virtcontainers subsystem=kata_agent Apr 01 22:59:58 u-01 containerd[17975]: time="2021-04-01T22:59:58.965535498Z" level=error msg="createContainer failed" error="rpc error: code = Internal desc =23:08
kata-irc-bot2unable to write to a control group file" pid=19838 sandbox=8aee44b3d558471182048faa704d805fc7bce25df77c1328e3f672d58054fe27 source=virtcontainers subsystem=kata_agent Apr 01 22:59:58 u-01 kata[19838]: time="2021-04-01T22:59:58.966757204Z" level=error msg="container create failed" error="rpc error: code = Internal desc = unable to write to a control group file" pid=19838 sandbox=8aee44b3d558471182048faa704d805fc7bce25df77c1328e3f672d58054fe2723:08
kata-irc-bot2source=virtcontainers subsystem=container Apr 01 22:59:58 u-01 kata[19838]: time="2021-04-01T22:59:58.966894184Z" level=warning msg="Could not umount" error="no such file or directory" host-path=/run/kata-containers/shared/sandboxes/8aee44b3d558471182048faa704d805fc7bce25df77c1328e3f672d58054fe27/mounts/6162629ffcc61c5d8e31e83adc901d9bfde611bdc8c3cf6fd873c8313c995108-480bde98affa440a-hosts pid=1983823:08
kata-irc-bot2sandbox=8aee44b3d558471182048faa704d805fc7bce25df77c1328e3f672d58054fe27 source=virtcontainers subsystem=container Apr 01 22:59:58 u-01 containerd[17975]: time="2021-04-01T22:59:58.966757204Z" level=error msg="container create failed" error="rpc error: code = Internal desc = unable to write to a control group file" pid=19838 sandbox=8aee44b3d558471182048faa704d805fc7bce25df77c1328e3f672d58054fe27 source=virtcontainers subsystem=container Apr 0123:08
kata-irc-bot222:59:58 u-01 containerd[17975]: time="2021-04-01T22:59:58.966894184Z" level=warning msg="Could not umount" error="no such file or directory" host-path=/run/kata-containers/shared/sandboxes/8aee44b3d558471182048faa704d805fc7bce25df77c1328e3f672d58054fe27/mounts/6162629ffcc61c5d8e31e83adc901d9bfde611bdc8c3cf6fd873c8313c995108-480bde98affa440a-hosts pid=19838 sandbox=8aee44b3d558471182048faa704d805fc7bce25df77c1328e3f672d58054fe2723:08
kata-irc-bot2source=virtcontainers subsystem=container Apr 01 22:59:58 u-01 containerd[17975]: time="2021-04-01T22:59:58.966968534Z" level=error msg="rollback failed unmountHostMounts()" error="no such file or directory" pid=19838 sandbox=8aee44b3d558471182048faa704d805fc7bce25df77c1328e3f672d58054fe27 source=virtcontainers subsystem=container Apr 01 22:59:58 u-01 containerd[17975]: time="2021-04-01T22:59:58.967051787Z" level=warning error="no such file or23:08
kata-irc-bot2directory" pid=19838 rootfs-dir=/run/kata-containers/shared/sandboxes/8aee44b3d558471182048faa704d805fc7bce25df77c1328e3f672d58054fe27/mounts/6162629ffcc61c5d8e31e83adc901d9bfde611bdc8c3cf6fd873c8313c995108/rootfs sandbox=8aee44b3d558471182048faa704d805fc7bce25df77c1328e3f672d58054fe27 source=virtcontainers subsystem=mount Apr 01 22:59:58 u-01 kata[19838]: time="2021-04-01T22:59:58.966968534Z" level=error msg="rollback failed unmountHostMounts()"23:08
kata-irc-bot2error="no such file or directory" pid=19838 sandbox=8aee44b3d558471182048faa704d805fc7bce25df77c1328e3f672d58054fe27 source=virtcontainers subsystem=container Apr 01 22:59:58 u-01 kata[19838]: time="2021-04-01T22:59:58.967051787Z" level=warning error="no such file or directory" pid=1983823:08
kata-irc-bot2rootfs-dir=/run/kata-containers/shared/sandboxes/8aee44b3d558471182048faa704d805fc7bce25df77c1328e3f672d58054fe27/mounts/6162629ffcc61c5d8e31e83adc901d9bfde611bdc8c3cf6fd873c8313c995108/rootfs sandbox=8aee44b3d558471182048faa704d805fc7bce25df77c1328e3f672d58054fe27 source=virtcontainers subsystem=mount```23:08
kata-irc-bot2<gmmaharaj> the issue happens with 9p as well. don't think virtiofs is the possible culprit here. I am testing on ubuntu 20.04 VM with nested enabled. ;)23:11
kata-irc-bot2<eric.ernst> That’s new to me. Any changes you make for the guest image, or for the runtime ?23:22
kata-irc-bot2<gmmaharaj> I am pulling 2.0.123:26
kata-irc-bot2<gmmaharaj> `curl https://raw.githubusercontent.com/kata-containers/kata-containers/2.0.1/tools/packaging/kata-deploy/kata-deploy/base/kata-deploy.yaml | sed 's/\(katadocker\/kata-deploy\)/\1:2.0.1/g' | kubectl apply -f -`23:26
kata-irc-bot2<gmmaharaj> that is how I applied it23:26
kata-irc-bot2<gmmaharaj> i am also on k8s 1.20.x. I am testing it now with 1.19.9 to see if it is k8s triggered or kata triggered23:27
kata-irc-bot2<gmmaharaj> same with k8s version 1.19.9 ```Apr 01 23:24:29 u-01 kata[8942]: time="2021-04-01T23:24:29.343186337Z" level=error msg="createContainer failed" error="rpc error: code = Internal desc = unable to write to a control group file" pid=8942 sandbox=adfeb5588bff84e3b54370dc8619ff7064b8183e20d1c0cf8896ee2f8f46f624 source=virtcontainers subsystem=kata_agent23:32
kata-irc-bot2                                                                                                                                         Apr 01 23:24:29 u-01 containerd[8377]: time="2021-04-01T23:24:29.343186337Z" level=error msg="createContainer failed" error="rpc error: code = Internal desc = unable to write to a control group file" pid=8942 sandbox=adfeb5588bff84e3b54370dc8619ff7064b8183e20d1c0cf8896ee2f8f46f624 source=virtcontainers23:32
kata-irc-bot2subsystem=kata_agent                                                                                                                                                                                                                            Apr 01 23:24:29 u-01 kata[8942]: time="2021-04-01T23:24:29.344381118Z" level=error msg="container create failed" error="rpc error: code = Internal desc = unable to write to a control group file" pid=894223:32
kata-irc-bot2sandbox=adfeb5588bff84e3b54370dc8619ff7064b8183e20d1c0cf8896ee2f8f46f624 source=virtcontainers subsystem=container  ```23:32
kata-irc-bot2<gmmaharaj> @eric.ernst launching the container directly using `ctr` works fine23:36
kata-irc-bot2<eric.ernst> crictl?23:38
kata-irc-bot2<eric.ernst> Something to do w/ pod is my guess.23:39
kata-irc-bot2<gmmaharaj> haven't tested iwth crictl23:39
kata-irc-bot2<eric.ernst> crictl run --runtime kata-qemu container.json pod.json23:39
kata-irc-bot2<eric.ernst> example ctr.json: ```{   "metadata": {     "name": "ubuntu"   },   "annotations": {   },   "image":{     "image": "docker.io/ubuntu:18.04"   },   "command": [     "/bin/sh"   ],   "args": [     "-c",     "while true; do /bin/date; sleep 5; done"   ],   "log_path":"ubuntu.log",   "linux": {   } }```23:40
kata-irc-bot2<eric.ernst> pd.json: ```{     "metadata": {         "name": "kata-sandbox",         "namespace": "default",         "attempt": 1,         "uid": "a-kata-uid"     },     "log_directory": "/tmp",     "linux": {     } }```23:41
kata-irc-bot2<gmmaharaj> thanks23:41
kata-irc-bot2<gmmaharaj> i just went down to 1.18.x23:41
kata-irc-bot2<eric.ernst> np.  crictl ux is kinda rough.23:41
kata-irc-bot2<eric.ernst> 1.19 should be fine.23:41
kata-irc-bot2<eric.ernst> this is somethign weird in the package or your setup.23:41
kata-irc-bot2<gmmaharaj> let me try it out later tonight and share some results23:42
kata-irc-bot2<gmmaharaj> have to step away from the PC now :(23:42
kata-irc-bot2<gmmaharaj> thanks for the yamls @eric.ernst23:43
« Wednesday, 2021-03-31 Index Friday, 2021-04-02 »

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!