| *** eernst has joined #kata-dev | 00:02 | |
| *** eernst has quit IRC | 00:06 | |
| *** eernst has joined #kata-dev | 00:08 | |
| *** eernst has quit IRC | 00:12 | |
| *** eernst has joined #kata-dev | 00:14 | |
| *** eernst has quit IRC | 00:17 | |
| kata-irc-bot | <mike> So.. question, what are you all doing to get real world metrics like cpu_time, etc into prometheus? I thought pointing cadvisor at crio/containerd.sock would do it but that seems to have its own huge limitations (crio has no identifying labels for instance).. any suggestions? | 00:34 |
|---|---|---|
| kata-irc-bot | <eric.ernst> hey @mike | 00:56 |
| kata-irc-bot | <eric.ernst> it won't work with the conmon method of interacting between CRIO and Kata. | 00:57 |
| kata-irc-bot | <eric.ernst> We have defined an improved interface which will facilitate utilizing direct calls to get container stats from the runtime | 00:57 |
| kata-irc-bot | <eric.ernst> with that in place, the kata-runtime gets the information over gRPC from inside the virtual machine and reports it back. | 00:58 |
| kata-irc-bot | <eric.ernst> Let me dig up the PR.... | 00:58 |
| kata-irc-bot | <mike> awesome, i think we stumbled across mention of that somewhere yesterday, thanks! | 00:58 |
| kata-irc-bot | <eric.ernst> see: https://github.com/kubernetes-sigs/cri-o/pull/2025 | 00:59 |
| kata-irc-bot | <eric.ernst> yep. @mike what do you work on? | 01:00 |
| kata-irc-bot | <eric.ernst> once that is in, all the metrics and metric driven work should be in good shape! It's pretty close; @sebastien.boeuf was showing it off this afternoon :slightly_smiling_face: | 01:01 |
| kata-irc-bot | <mike> we’re testing out running untrusted workloads with kata + crio.. have to be a bit vague right now | 01:11 |
| kata-irc-bot | <mike> @eric.ernst would this PR mean that ContainerStatsRequest (from cri) generated by kubelet should get metrics directly from a kata container? still digging through it | 01:12 |
| kata-irc-bot | <mike> Like kubelet -> cri-o -> kata_runtime -> kata_agent, and basically bubble the metrics all the way back up from the container | 01:13 |
| kata-irc-bot | <sebastien.boeuf> It will return the stats as they're returned by the kata runtime, instead of using cadvisor by assuming the container is running on the host | 01:13 |
| kata-irc-bot | <sebastien.boeuf> Yes exactly as you described it. Down from inside the VM up to the CRI | 01:14 |
| kata-irc-bot | <eric.ernst> No worries. Let us know if we can help clarify anything. | 01:15 |
| kata-irc-bot | <xwlpt> @eric.ernst I found that this issue only happens on the VM that hv’s cpu is skylake, but it is fine for Haswell CPU. | 01:17 |
| kata-irc-bot | <xwlpt> I think there is another issue, when this happens, qemu process will not exit, then there will be more and more qemu process running on that host, it will caused the host can’t work after some time. | 01:18 |
| kata-irc-bot | <eric.ernst> Can you share in detail how you grabbed the log? | 01:19 |
| kata-irc-bot | <sebastien.boeuf> Why? | 01:19 |
| kata-irc-bot | <sebastien.boeuf> I don't think you get how the whole stack work, there's no reason for QEMU to stick around | 01:19 |
| kata-irc-bot | <eric.ernst> I saw same when I had same symptom. | 01:20 |
| kata-irc-bot | <eric.ernst> It stays, and Kubernetes tried again. | 01:20 |
| kata-irc-bot | <eric.ernst> Error back off, each one a new QEMU instance. | 01:20 |
| kata-irc-bot | <sebastien.boeuf> Oh sorry I thought you were talking about the actual PR | 01:20 |
| kata-irc-bot | <eric.ernst> This was when I was running kata as L3 hypervisor thiugg. | 01:20 |
| kata-irc-bot | <eric.ernst> You’re seeing in L2? | 01:20 |
| kata-irc-bot | <xwlpt> ``` | 01:23 |
| kata-irc-bot | <xwlpt> change the configuration.toml [hypervisor.qemu] path = “/path/to/qemu/kata-qemu-wrapper” to a script. And insdie the scrpit, it runs: /home/qemu/x86_64-softmmu/qemu-system-x86_64 “$@” -D /tmp/qemu.log | 01:23 |
| kata-irc-bot | <xwlpt> @eric.ernst | 01:23 |
| kata-irc-bot | <xwlpt> Then you can see the qemu log at the /tmp | 01:23 |
| kata-irc-bot | <eric.ernst> Yeah, that’s what I did too | 01:23 |
| kata-irc-bot | <eric.ernst> Okay; I just didn’t anything written, unfortunately. | 01:24 |
| kata-irc-bot | <xwlpt> @eric.ernst So the VM had launched successfully? | 01:25 |
| kata-irc-bot | <xwlpt> If so , it should have console log | 01:25 |
| kata-irc-bot | <xwlpt> You can dump the boot logs too | 01:25 |
| kata-irc-bot | <eric.ernst> Change kernel Cmdline or anything for this? | 01:25 |
| kata-irc-bot | <xwlpt> add console=ttyS0 in configuration.toml’s kernel_params= | 01:26 |
| kata-irc-bot | <xwlpt> Add ` -serial file:/tmp/serial.log` to the script | 01:26 |
| kata-irc-bot | <mike> any idea what release that #2025 PR is targeting? curious on time frame, not to be pushy | 01:26 |
| kata-irc-bot | <mike> 1.6? | 01:26 |
| kata-irc-bot | <eric.ernst> Thx. Will try this later tonight. | 01:26 |
| kata-irc-bot | <eric.ernst> Are you using vsock, by any chance? | 01:27 |
| kata-irc-bot | <xwlpt> I am using it now. But this issue also happens when it use kata-proxy | 01:28 |
| kata-irc-bot | <eric.ernst> The changes are actually part of CRI-O, not Kata | 01:35 |
| kata-irc-bot | <mike> oh i totally missed that sorry | 01:36 |
| kata-irc-bot | <eric.ernst> @sebastien.boeuf can clarify, but I’m pretty sure no changes are required on kata side at this point. | 01:43 |
| *** dklyle has quit IRC | 01:46 | |
| *** david-lyle has joined #kata-dev | 01:46 | |
| *** dklyle has joined #kata-dev | 01:48 | |
| *** david-lyle has quit IRC | 01:50 | |
| *** dims has quit IRC | 02:53 | |
| *** igordc has quit IRC | 04:32 | |
| *** stackedsax has quit IRC | 04:52 | |
| kata-irc-bot | <raravena80> yeah, after the PR you should see the metrics on the kubelet /metrics endpoint | 05:26 |
| kata-irc-bot | <raravena80> this seems relevant: https://github.com/kubernetes/kubernetes/pull/73517 | 05:30 |
| kata-irc-bot | <eric.ernst> yeah, i've been watching that.... not sure its the best solution as is, but good start to the conversation? | 05:32 |
| kata-irc-bot | <raravena80> I think so. More information perhaps how Kata can enhance multi-tenancy. | 05:44 |
| kata-irc-bot | <sebastien.boeuf> @eric.ernst is right, nothing is needed from the kata side. The PR #2025 from CRIO should be enough to get V2 working. The kata-v2 binary is shared between containerd and CRIO implementations. | 06:41 |
| *** sameo has joined #kata-dev | 08:44 | |
| *** jodh has joined #kata-dev | 08:58 | |
| *** davidgiluk has joined #kata-dev | 09:07 | |
| *** eguan has quit IRC | 09:21 | |
| *** eguan has joined #kata-dev | 09:22 | |
| *** gwhaley has joined #kata-dev | 09:33 | |
| *** gwhaley has quit IRC | 12:01 | |
| *** devimc has joined #kata-dev | 13:15 | |
| *** gwhaley has joined #kata-dev | 13:22 | |
| *** lpetrut has joined #kata-dev | 13:28 | |
| *** dims has joined #kata-dev | 13:41 | |
| *** dims has quit IRC | 14:38 | |
| *** dims has joined #kata-dev | 14:44 | |
| *** dims has quit IRC | 15:01 | |
| *** eernst has joined #kata-dev | 15:23 | |
| *** lpetrut has quit IRC | 15:24 | |
| *** eernst has quit IRC | 15:25 | |
| *** eernst has joined #kata-dev | 15:25 | |
| *** dims has joined #kata-dev | 15:55 | |
| *** lpetrut has joined #kata-dev | 16:00 | |
| *** eernst has quit IRC | 16:07 | |
| *** sameo has quit IRC | 16:27 | |
| *** lpetrut has quit IRC | 16:33 | |
| *** eernst has joined #kata-dev | 16:36 | |
| *** eernst has quit IRC | 16:39 | |
| *** lpetrut has joined #kata-dev | 16:46 | |
| *** igordc has joined #kata-dev | 16:59 | |
| *** stackedsax has joined #kata-dev | 17:05 | |
| *** lpetrut has quit IRC | 17:45 | |
| *** lpetrut has joined #kata-dev | 17:46 | |
| *** jodh has quit IRC | 17:52 | |
| *** gwhaley has quit IRC | 18:12 | |
| *** lpetrut has quit IRC | 18:29 | |
| *** fdegir has quit IRC | 18:38 | |
| *** fdegir has joined #kata-dev | 18:39 | |
| *** devimc_ has joined #kata-dev | 19:31 | |
| *** devimc has quit IRC | 19:33 | |
| *** devimc__ has joined #kata-dev | 20:07 | |
| *** devimc_ has quit IRC | 20:10 | |
| *** eernst has joined #kata-dev | 20:14 | |
| *** eernst has quit IRC | 20:14 | |
| *** eernst has joined #kata-dev | 20:16 | |
| *** eernst has quit IRC | 20:16 | |
| *** eernst has joined #kata-dev | 20:16 | |
| *** devimc__ has quit IRC | 20:27 | |
| *** devimc has joined #kata-dev | 20:30 | |
| *** m3thos has joined #kata-dev | 20:45 | |
| *** davidgiluk has quit IRC | 20:49 | |
| kata-irc-bot | <eric.ernst> @eric.adams WDYT? | 21:49 |
| kata-irc-bot | <eric.ernst> @manohar.r.castelino and @archana.m.shinde we should take a look at our library of ascii vids | 21:49 |
| *** devimc has quit IRC | 22:04 | |
| *** eernst has quit IRC | 22:48 | |
| *** eernst has joined #kata-dev | 22:51 | |
| *** eernst has quit IRC | 22:53 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!