| kata-irc-bot | <fidencio> @fupan: ```Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 29s default-scheduler Successfully assigned default/nginx-deployment-qemu-5d97b689f4-r4j4x to k8s-agentpool-16181271-0 Warning FailedCreatePodSandBox 5s (x3 over 28s) kubelet Failed to create pod | 07:03 |
|---|---|---|
| kata-irc-bot | sandbox: rpc error: code = NotFound desc = failed to create containerd task: not found``` Does this ring a bell for you? Can it be related to using a too old version of containerd? | 07:03 |
| kata-irc-bot | <fupan> It seemed the shimv2 didn’t run, was there any more logs? | 07:06 |
| kata-irc-bot | <fidencio> @fupan, not much apart from that, really. ```level=info msg="RunPodsandbox for &PodSandboxMetadata{Name:nginx-deployment-qemu-5d97b689f4-fp6wt,Uid:f6f32e42-6e06-4d97-b660-0c7bf158cf04,Namespace:default,Attempt:0,}" level=debug msg="Sandbox config &PodSandboxConfig{Metadata:&PodSandboxMetadata{Name:nginx-deployment-qemu-5d97b689f4-fp6wt,Uid:f6f32e42-6e06-4d97-b660-0c7bf158cf04,Namespace: level=debug msg="Generated id | 07:46 |
| kata-irc-bot | \"35679da66a5f08c7ebd456032b3500042cfe3fd2343acf8c7a5bf8265a501ee2\" for sandbox \"nginx-deployment-qemu-5d97b689f4-fp6wt_default_f6f32e42-6e06- level=debug msg="Use OCI {Type:io.containerd.kata-qemu.v2 Engine: PodAnnotations:[io.katacontainers.*] ContainerAnnotations:[] Root: Options:0xc00042ca20 PrivilegedWithoutHos level=debug msg="cni result for sandbox \"35679da66a5f08c7ebd456032b3500042cfe3fd2343acf8c7a5bf8265a501ee2\": | 07:46 |
| kata-irc-bot | {\"Interfaces\":{\"eth0\":{\"IPConfigs\":[{\"IP\":\"10.240.0.15\ level=debug msg="Sandbox container \"35679da66a5f08c7ebd456032b3500042cfe3fd2343acf8c7a5bf8265a501ee2\" spec: (*specs.Spec)(0xc0005da680){Version:(string)1.0.2-dev Process:(* level=debug msg="event published" ns=k8s.io topic=/snapshot/prepare type=containerd.events.SnapshotPrepare level=debug msg="event published" ns=k8s.io topic=/containers/create | 07:46 |
| kata-irc-bot | type=containerd.events.ContainerCreate level=debug msg="registering ttrpc server" level=debug msg="serving api on socket" socket="[inherited from parent]" level=info msg="starting signal loop" namespace=k8s.io path=/run/containerd/io.containerd.runtime.v2.task/k8s.io/35679da66a5f08c7ebd456032b3500042cfe3fd2343acf8c7a5bf8265a501 level=debug msg="converting | 07:46 |
| kata-irc-bot | /run/containerd/io.containerd.runtime.v2.task/k8s.io/35679da66a5f08c7ebd456032b3500042cfe3fd2343acf8c7a5bf8265a501ee2/config.json" name=containerd level=error msg="trace called before context set" name=containerd-shim-v2 pid=1467 sandbox=35679da66a5f08c7ebd456032b3500042cfe3fd2343acf8c7a5bf8265a501ee2 source=containerd- level=debug msg="failed to delete task" error="rpc error: code = NotFound desc = container does not exist | 07:46 |
| kata-irc-bot | 35679da66a5f08c7ebd456032b3500042cfe3fd2343acf8c7a5bf8265a501ee2: no``` | 07:46 |
| kata-irc-bot | <fidencio> (sorry for the cropped output) | 07:46 |
| kata-irc-bot | <fidencio> ```# containerd --version containerd github.com/containerd/containerd 1.4.4+azure 05f951a3781f4f2c1911b05e61c160e9c30eaa8e``` | 07:48 |
| kata-irc-bot | <fidencio> @fupan, it looks like https://github.com/kata-containers/kata-containers/issues/1971#issuecomment-856024439 | 07:50 |
| kata-irc-bot | <fidencio> But on the opposite direction | 07:50 |
| kata-irc-bot | <fidencio> and I can't bump to a newer version of containerd to test, as aks tells me: ```please use one of the following versions: [1.3.2 1.3.3 1.3.4 1.3.5 1.3.6 1.3.7 1.3.8 1.3.9 1.4.4``` | 07:52 |
| kata-irc-bot | <fidencio> And I can't switch to CRI-O either (which was a workaround that crossed my mind, just to get the tarballs released) | 07:53 |
| kata-irc-bot | <fupan> Hmm, it’s really seemed that the kata shimv2 didn’t match the containerd. BTW, what’s the kata shim’s version ? | 09:23 |
| kata-irc-bot | <fidencio> That's main | 09:24 |
| kata-irc-bot | <fidencio> The issue I'm facing is in the GitHub action for releasing the tarball. | 09:25 |
| kata-irc-bot | <fidencio> The action gets the content that will be released, deploys an AKS cluster for us (which happens to use that version of containerd, and that's the newest one we can have there), and tries to run some pods | 09:25 |
| kata-irc-bot | <fidencio> @fupan, do we need to change something in the containerd configuration file or that shouldn't be affected at all? | 09:27 |
| kata-irc-bot | <fidencio> (I don't think we do, but just double-checking) | 09:28 |
| kata-irc-bot | <fupan> Yeah, containerd’s config couldn’t affect it. But I’m curious, I had just test the main branch against the 1.4.4 containerd ” revision=05f951a3781f4f2c1911b05e61c160e9c30eaa8e version=v1.4.4" , but it worked well. | 09:33 |
| kata-irc-bot | <fidencio> That's interesting. I've started some tests in the CI, where we have more control and can tweak better the logs we get, in order to see if I can reproduce the issue there as well | 09:36 |
| kata-irc-bot | <fidencio> https://github.com/kata-containers/kata-containers/pull/2255 | 09:36 |
| kata-irc-bot | <fupan> It’s weird, I saw this error info in your log: level=error msg=“trace called before context set” name=containerd-shim-v2 pid=1467 | 09:41 |
| kata-irc-bot | <fupan> But my commits f607641a6e6d8240c2f82484fb8ef7d4d01151de had changed it, thus I guess your case didn’t use the main branch | 09:41 |
| kata-irc-bot | <fidencio> It's using the main branch ... well, it's using kata-deploy from https://github.com/kata-containers/kata-containers/commit/594ff3a5bde3813f29ee2a565e0526b35702be4a | 09:43 |
| kata-irc-bot | <fupan> But I had just recheck the main branch, and it did’t output the error log of “trace called before context set” | 09:55 |
| kata-irc-bot | <fidencio> ```# ./containerd-shim-kata-v2 --version Kata Containers containerd shim: id: "io.containerd.kata.v2", version: 2.2.0-alpha1, commit: 594ff3a5bde3813f29ee2a565e0526b35702be4a``` | 09:55 |
| kata-irc-bot | <fupan> So you opened the trace function ? | 10:00 |
| kata-irc-bot | <fidencio> @fupan, I sincerely think so, as it's using 594ff3a5bde3813f29ee2a565e0526b35702be4a and that commit has yours included | 10:06 |
| kata-irc-bot | <fupan> So can you paste the kata configure file and let me have a try? | 10:07 |
| kata-irc-bot | <fidencio> https://paste.centos.org/view/cacfd04b | 10:10 |
| kata-irc-bot | <fidencio> And I could reproduce the same on our CI: http://jenkins.katacontainers.io/job/kata-containers-2.0-ubuntu-20.04-PR/503/console | 10:26 |
| kata-irc-bot | <fidencio> The tracing message is just a red herring | 12:38 |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!