Tuesday, 2020-12-01

« Monday, 2020-11-30 Index Wednesday, 2020-12-02 »
*** th0din has quit IRC00:35
*** devimc has quit IRC00:44
*** th0din has joined #kata-dev00:46
*** Yarboa has quit IRC01:03
*** Yarboa has joined #kata-dev01:05
*** Yarboa has quit IRC02:03
*** Yarboa has joined #kata-dev02:04
*** Yarboa has quit IRC03:03
*** Yarboa has joined #kata-dev03:03
*** crobinso has quit IRC03:22
*** Yarboa has quit IRC03:54
*** Yarboa has joined #kata-dev03:54
*** sameo has joined #kata-dev05:08
*** Yarboa has quit IRC06:03
*** Yarboa has joined #kata-dev06:03
*** dklyle has quit IRC07:42
*** pcaruana has joined #kata-dev07:46
*** ailan has joined #kata-dev08:05
*** fgiudici has joined #kata-dev08:17
*** sgarzare has joined #kata-dev08:27
*** fgiudici has quit IRC09:02
*** fgiudici_ has joined #kata-dev09:02
*** Yarboa has quit IRC09:03
*** davidgiluk has joined #kata-dev09:03
*** Yarboa has joined #kata-dev09:15
kata-irc-bot<fidencio> @bergwolf, good evening!  So, I'm facing `containerd-shim-kata-v2`  processes hanging around when after `kubectl delete pod ...`, causing the pods to be always in a `Terminating` state.10:20
kata-irc-bot<fidencio> Any pointer on where to look?10:21
kata-irc-bot<fidencio> The last kata log I received was that vsock communication failed (and that's expected as QEMU is off).10:21
kata-irc-bot<bergwolf> is the shim v2 process still running?10:24
kata-irc-bot<bergwolf> it is supposed to stop if qemu is gone10:24
kata-irc-bot<fidencio> Yes, the shimv2 process is the one that keeps running10:26
kata-irc-bot<fidencio> There's a weird message after qemu goes down about "New client" (still need to investigate that)10:26
kata-irc-bot<bergwolf> then maybe gdb attach and see what it is doing?10:27
kata-irc-bot<bergwolf> then maybe gdb attach and see what it is doing?10:27
kata-irc-bot<bergwolf> after qemu is dead, shimv2 process should find it out and quit itself10:27
kata-irc-bot<fidencio> Ack!10:31
*** ailan has quit IRC10:47
kata-irc-bot<fidencio> https://paste.centos.org/view/0c5b13fd Seems it's just busy waiting for something (some notification that the connection was closed on the other side?)10:58
kata-irc-bot<fidencio> Okay, okay, that's material for the afternoon debugging.10:58
kata-irc-bot<liubin0329> Had there any exec operations before delete the pod? I found exec may lead deleting pod to hang before.11:01
kata-irc-bot<fidencio> Hey @liubin0329, no no exec operations before deleting the pod.11:01
kata-irc-bot<fidencio> Argh, foreget about the pastebin, I need more coffee. :slightly_smiling_face:11:04
kata-irc-bot<fidencio> That seems to be the reason: ```Thread 8 (Thread 0x7fe732b1a700 (LWP 59673)): #0  syscall.Syscall () at /usr/lib/golang/src/syscall/asm_linux_amd64.s:24 #1  0x000055d37ead0305 in golang.org/x/sys/unix.Close (fd=3, err=...)     at /builddir/build/BUILD/kata-containers-2.0-gb8716d8e/go/src/%{importname}/src/runtime/vendor/golang.org/x/sys/unix/zsyscall_linux.go:509 #2  0x000055d37f3494f0 in11:06
kata-irc-botgithub.com/mdlayher/vsock.(*sysConnFD).EarlyClose (cfd=0xc000522330, ~r0=...)     at /builddir/build/BUILD/kata-containers-2.0-gb8716d8e/go/src/%{importname}/src/runtime/vendor/github.com/mdlayher/vsock/fd_linux.go:137 #3  0x000055d37f34b888 in github.com/mdlayher/vsock.dialLinux.func1 (&err=0xc000477ea8, cfd=...)     at11:06
kata-irc-bot/builddir/build/BUILD/kata-containers-2.0-gb8716d8e/go/src/%{importname}/src/runtime/vendor/github.com/mdlayher/vsock/conn_linux.go:41 #4  0x000055d37f349365 in github.com/mdlayher/vsock.dialLinux (cfd=..., cid=1177951647, port=1024, c=0x0, err=...)     at /builddir/build/BUILD/kata-containers-2.0-gb8716d8e/go/src/%{importname}/src/runtime/vendor/github.com/mdlayher/vsock/conn_linux.go:51 #5  0x000055d37f349047 in github.com/mdlayher/vsock.dial11:06
kata-irc-bot(cid=1177951647, port=1024, ~r2=<optimized out>, ~r3=...)     at /builddir/build/BUILD/kata-containers-2.0-gb8716d8e/go/src/%{importname}/src/runtime/vendor/github.com/mdlayher/vsock/conn_linux.go:32 #6  0x000055d37f34a1d7 in github.com/mdlayher/vsock.Dial (contextID=1177951647, port=1024, ~r2=<optimized out>, ~r3=...)     at11:06
kata-irc-bot/builddir/build/BUILD/kata-containers-2.0-gb8716d8e/go/src/%{importname}/src/runtime/vendor/github.com/mdlayher/vsock/vsock.go:140 #7  0x000055d37f34d82c in github.com/kata-containers/kata-containers/src/runtime/virtcontainers/pkg/agent/protocols/client.VsockDialer.func1 (~r0=..., ~r1=...)     at /builddir/build/BUILD/kata-containers-2.0-gb8716d8e/go/src/%{importname}/src/runtime/virtcontainers/pkg/agent/protocols/client/client.go:288 #811:06
kata-irc-bot0x000055d37f34d738 in github.com/kata-containers/kata-containers/src/runtime/virtcontainers/pkg/agent/protocols/client.commonDialer.func1 (cancel=0xc00008a1e0, dialFunc={void (net.Conn *, error *)} 0xc000477fc8, t=0xc000088690, ch=0xc00008a2a0)     at /builddir/build/BUILD/kata-containers-2.0-gb8716d8e/go/src/%{importname}/src/runtime/virtcontainers/pkg/agent/protocols/client/client.go:252 #9  0x000055d37ea1d2e1 in runtime.goexit () at11:06
kata-irc-bot/usr/lib/golang/src/runtime/asm_amd64.s:1374```11:06
*** snir has quit IRC12:34
*** snir has joined #kata-dev12:36
*** devimc has joined #kata-dev13:05
*** amorenoz has quit IRC13:07
*** pcaruana has quit IRC14:03
*** fuentess has joined #kata-dev14:13
*** pcaruana has joined #kata-dev14:14
*** pcaruana has quit IRC15:01
*** pcaruana has joined #kata-dev15:02
*** crobinso has joined #kata-dev15:19
*** pcaruana has quit IRC15:36
*** pcaruana has joined #kata-dev15:49
*** irclogbot_3 has quit IRC15:53
*** irclogbot_1 has joined #kata-dev15:55
*** devimc has quit IRC15:56
*** devimc has joined #kata-dev15:56
*** dklyle has joined #kata-dev15:57
*** irclogbot_1 has quit IRC16:11
*** irclogbot_3 has joined #kata-dev16:11
fidenciodevimc: do we use image as default on kata-containers 2.0?16:52
fidenciodevimc: didn't we fallback to using initrd?16:52
fidenciodevimc: and buenos dias! :-)16:52
devimcfidencio, bom dia, yes, but afaik both can't be specified at the same time16:54
fidenciodevimc: but what's the default on 2.x? image?17:00
devimcyes17:00
fidenciodevimc: ack, thanks!17:00
kata-irc-bot<eric.ernst> Anyone else see this? https://github.com/kata-containers/kata-containers/issues/116017:33
kata-irc-bot<eric.ernst> I haven't seen this @fidencio -- this with tip? A release?17:34
kata-irc-bot<eric.ernst> and can you reproduce 100% of time?17:35
kata-irc-bot<fidencio> I can reproduce this 100% of the time, it's with `git main` , on `2.0-dev` branch running on OCP 4.6.5.17:35
kata-irc-bot<fidencio> I'll get more info and have it fixed as I have the feeling it may be related to the our own environment /o\17:36
kata-irc-bot<fidencio> I'm not sure if I can reproduce the same on CentOS, but that's something I'm trying right now.17:37
kata-irc-bot<fidencio> What I'm facing is quite similar, Eric.17:41
kata-irc-bot<eric.ernst> okay.  I haven't seen this at all w/ upstream kubernetes + containerd17:43
kata-irc-bot<eric.ernst> this is only with exec  with backgroudn process left behind, though.17:48
kata-irc-bot<eric.ernst> Works fine otherwise.17:48
*** fgiudici_ has quit IRC18:17
kata-irc-bot<fidencio> Okay, this is OCP specific!  I simply can't reproduce the issue when using a CentOS 8 VM serving as a single node k8s cluster, running latest `release-1.18`  CRI-O's branch.18:17
kata-irc-bot<fidencio> Let me see if I can easily reproduce this here18:18
kata-irc-bot<fidencio> yeah, I can easily reproduce this with CRI-O as well.18:20
*** sgarzare has quit IRC19:03
*** crobinso has quit IRC19:42
*** davidgiluk has quit IRC20:09
*** pcaruana has quit IRC20:27
*** sameo has quit IRC20:53
*** devimc has quit IRC22:06
*** sameo has joined #kata-dev22:36
*** crobinso has joined #kata-dev22:48
« Monday, 2020-11-30 Index Wednesday, 2020-12-02 »

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!