| *** eernst has quit IRC | 00:03 | |
| *** igordc has quit IRC | 01:35 | |
| *** mugsie has quit IRC | 03:22 | |
| *** mugsie has joined #kata-dev | 03:37 | |
| *** irclogbot_2 has quit IRC | 03:41 | |
| *** irclogbot_1 has joined #kata-dev | 03:41 | |
| *** igordc has joined #kata-dev | 04:12 | |
| *** igordc has quit IRC | 04:19 | |
| *** pcaruana has joined #kata-dev | 05:09 | |
| *** tmhoang has joined #kata-dev | 07:16 | |
| *** jodh has joined #kata-dev | 07:30 | |
| kata-irc-bot | <james.o.hunt> @eric.ernst - Morning - could you comment on https://github.com/kata-containers/packaging/pull/572 please? | 07:33 |
|---|---|---|
| kata-irc-bot | <james.o.hunt> @eric.ernst - nm - I've been a victim of browser caching again! ;( | 07:35 |
| *** sameo has joined #kata-dev | 07:36 | |
| *** gwhaley has joined #kata-dev | 07:43 | |
| *** davidgiluk has joined #kata-dev | 08:03 | |
| kata-irc-bot | <niteshkonkar007> Hello! I am unable to connect to the Kata VM bash shell, post following the developer guide steps. Any pointers what could be missing ... ``` [root@nitkon image-builder]# sudo socat -v "stdin,raw,echo=0,escape=0x11" "unix-connect:${console}" > 2019/06/14 13:37:12.256090 length=1 from=0 to=0 \r< 2019/06/14 13:37:12.256777 length=2 from=0 to=1 ``` | 08:10 |
| kata-irc-bot | \r | 08:10 |
| stefanha | niteshkonkar007: It can be useful to add "-serial /tmp/serial.log" to the QEMU command-line and "console=ttyS0" to the kernel parameters. That way you get console output and can see how far the sandbox VM booted. (This is not an interactive terminal, just output.) | 08:17 |
| stefanha | niteshkonkar007: Are you using a systemd-based rootfs? If you're using an initramfs without systemd then the debug console won't come up because only kata-agent is running, not systemd. | 08:17 |
| kata-irc-bot | <niteshkonkar007> Correct. So I am on x86, running a rootfs systemd based image. Two questions here: 1. How can we achieve the same for initrd based non-systemd image? 2. I assumed the sandbox VM has booted completely, coz I could exec inside the kata container. Am I wrong? | 08:20 |
| kata-irc-bot | <graham.whaley> @stefanha - ooi, are those serial/log hints in our kata docs? If not, any chance you could PR an addition? ;) | 08:40 |
| stefanha | niteshkonkar007: 1. If kata-agent is running as pid 1 (no systemd) then I'm not aware of a way to spawn a debug console. Best to stick to a systemd rootfs for debugging... 2. Yes, sounds like the sandbox VM is booting successfully if you can run a container. | 08:42 |
| stefanha | graham.whaley: Sure, I'll update the developer guide | 08:42 |
| kata-irc-bot | <niteshkonkar007> @stefanha: PPC as of today only supports the initrd based approach. Hence I wanted to know. But I shifted to x86 for debugging purpose using rootfs systemd approach. | 08:46 |
| brtknr | Hmm what happened to this issue? https://github.com/kata-containers/runtime/issues/356 | 08:49 |
| stefanha | niteshkonkar007: I have run an initramfs with systemd, so you could use that for debugging on ppc. I think kata-agent just needed a little tweak to not use pivot_root(2) in this configuration. But my understanding is that initramfs+systemd isn't widely used and isn't supported in Kata. | 08:51 |
| stefanha | niteshkonkar007: osbuilder does let you build an initramfs with systemd. I used the Fedora image. | 08:51 |
| kata-irc-bot | <graham.whaley> morning @salvador.fuentes @eric.ernst - ooi, I just did a very quick one-look check to see if our CI optimisations may have any visible effect. Just looking at the latest runtime 18.04 builds, and one from a week ago .... first look says we have gone from a 1h43m build to a 1h2m build. Sure, we should check other data, but first look that is a fine improvement :slightly_smiling_face: | 08:53 |
| kata-irc-bot | <wei> @niteshkonkar007 with initrd image, you can add “agent.debug_console” to kernel parameters to activate the debug console | 08:57 |
| kata-irc-bot | <bergwolf> `agent.debug_console` works for both rootfs and initrd images. kata agent spawns a debug console when it is specified. | 09:03 |
| stefanha | nice! :) | 09:03 |
| kata-irc-bot | <niteshkonkar007> Thanks! Adding `agent.debug_console` to `kernel_params` worked! :slightly_smiling_face: | 09:11 |
| kata-irc-bot | <graham.whaley> @niteshkonkar007 - excellent - pls check that is in the docs already, and add if not :slightly_smiling_face: | 09:12 |
| kata-irc-bot | <niteshkonkar007> @graham.whaley: Sure. | 09:15 |
| kata-irc-bot | <penny.zheng> out of curiosity, if i choose vsock, the debug info in kata will be sent to where? | 10:13 |
| brtknr | is the debug console read only? | 10:50 |
| brtknr | opened using socat? | 10:50 |
| kata-irc-bot | <niteshkonkar007> It is not read only & is opened using socat | 10:55 |
| *** tmhoang has quit IRC | 11:01 | |
| kata-irc-bot | <graham.whaley> it is, however, disabled by default as a security measure ;) | 11:09 |
| brtknr | hmm, wondering why I dont see anything in my console... i placed agent.debug_console param in my kernel_params | 11:11 |
| kata-irc-bot | <niteshkonkar007> & `EXTRA_PKGS="bash coreutils"` when building rootfs? | 11:14 |
| *** tmhoang has joined #kata-dev | 12:12 | |
| *** tmhoang has quit IRC | 12:18 | |
| *** tmhoang has joined #kata-dev | 12:41 | |
| *** tmhoang has quit IRC | 12:49 | |
| *** fuentess has joined #kata-dev | 12:51 | |
| *** dklyle has quit IRC | 14:42 | |
| *** dklyle has joined #kata-dev | 14:42 | |
| *** lpetrut has joined #kata-dev | 14:46 | |
| brtknr | is it possible to use the existing kata image with the debug_console option by modifying the toml file? | 14:48 |
| brtknr | that comes with kata-deploy | 14:48 |
| *** sameo has quit IRC | 14:50 | |
| *** eernst has joined #kata-dev | 14:54 | |
| kata-irc-bot | <eric.ernst> shouldb e. | 14:56 |
| kata-irc-bot | <graham.whaley> @brtknr: I think we recently optimised the image to reduce what was installed, and that included now not installing a shell. so, I think not (OK, technically you do connect to the console, but there is nothing there to 'chat' to). See nitkons PR today, that talks about adding the shell package back in... | 14:56 |
| brtknr | gwhaley: thanks | 15:00 |
| kata-irc-bot | <eric.ernst> @sebastien.boeuf you around? | 16:33 |
| eernst | davidgiluk: around? | 16:33 |
| eernst | (hope not - happy friday!) | 16:33 |
| davidgiluk | /me nod | 16:33 |
| davidgiluk | eernst: How may I be of service? | 16:33 |
| eernst | I haven't been tracking some of the outstanding work around virtio-fs very well, and wanted to better understand hotplug | 16:33 |
| eernst | ie, if / when /ever memory hotplug would be feasible. | 16:34 |
| eernst | Without it, we'll need to get creative for users who would like to utilize virito-fs + Kata | 16:34 |
| davidgiluk | eernst: My understanding is that it should be, but I've not tried it | 16:35 |
| davidgiluk | eernst: As long as the memory that's hot plugged also has the share=on flag it's got a reasonable chance | 16:35 |
| stefanha | What's the best way for us to test memory hotplug through Kata? | 16:36 |
| eernst | Hey stefanha -- in digging up the issue for davidgiluk i now see there's been some dialogue | 16:36 |
| eernst | https://github.com/kata-containers/runtime/issues/1745 | 16:36 |
| davidgiluk | eernst: The theory is that qemu should send a new set_mem_table message to the virtiofsd telling it about the extra memory, the libvhost-user should take care of it and all just magically works - right? | 16:36 |
| eernst | docker is probably easiest, unless you happen to be running a k8s based system. | 16:37 |
| eernst | Just specify memory limits when running kata-nemu | 16:37 |
| davidgiluk | it might be best if we give it a try first outside kata and see if it works | 16:37 |
| davidgiluk | eernst: Can I ask how many times do you tend to hotplug - is this once near startup or lots fo times? | 16:38 |
| stefanha | davidgiluk: Yep, in theory that's how vhost-user should handle memory hotplug. | 16:38 |
| davidgiluk | stefanha: I don't think I quite understand how that actually works in practice when we were doing a read() into an old mapping | 16:40 |
| stefanha | davidgiluk: There is a race if one thread processes vhost-user messages while the other handles virtqueue kicks. Is that what you mean? | 16:41 |
| kata-irc-bot | <sebastien.boeuf> @eric.ernst yes | 16:41 |
| kata-irc-bot | <sebastien.boeuf> what's up? | 16:41 |
| eernst | stefanha: assuming you have docker already installed... see https://github.com/kata-containers/runtime/issues/1745#issuecomment-502181598 | 16:41 |
| davidgiluk | stefanha: Yes I think so - it depends how smart it is about adding new mappings | 16:41 |
| eernst | that's the quickest way to reproduce | 16:41 |
| eernst | show a passing case (no hotplug) and failing case (hotplug) | 16:42 |
| *** sameo has joined #kata-dev | 16:48 | |
| kata-irc-bot | <sebastien.boeuf> eernst: well that's expected | 16:50 |
| kata-irc-bot | <sebastien.boeuf> unless the virtiofs daemon is updated with support for the update of the memory regions, this will not work | 16:51 |
| *** gwhaley has quit IRC | 16:51 | |
| kata-irc-bot | <sebastien.boeuf> davidgiluk: yeah exactly, given the fact that QEMU is able to resend a new set_mem_table, then it's up to the daemon to update its internal mapping | 16:52 |
| davidgiluk | seb: Well, there's code in vu_set_mem_table_exec that unmaps old ones and maps the new ones | 16:54 |
| davidgiluk | seb: I'm just not sure I trust it to do it while the device is in use | 16:54 |
| stefanha | I can take a look at this on Monday. In theory virtiofsd can do it but it hasn't been tested. I'll verify that the new vhost-user set_mem_table message is being sent and that it gets handled. | 16:54 |
| kata-irc-bot | <sebastien.boeuf> oh that's cool I didn't see this was in the daemon's code | 16:54 |
| kata-irc-bot | <sebastien.boeuf> davidgiluk: why don't you trust the daemon for doing the update while the device is in use? | 16:55 |
| davidgiluk | seb: It might have to be a bit smarter - the trick in theory with a hotplug is that you should only be adding a new region, so you should be able to keep all the other regions the same and it's all perfectly safe because the regions in use dont change | 16:55 |
| davidgiluk | seb: Well, for a start, it unmap's everything and then maps the new ones - that suggests there's a gap where nothing is mapped | 16:55 |
| kata-irc-bot | <sebastien.boeuf> oh yes that's even better if you only accept new regions | 16:55 |
| stefanha | davidgiluk: libvhost-user should probably have an API for marking the quiescent point in virtqueue processing threads (kind of like RCU). Then it the mappings can be replaced safely. | 16:56 |
| eernst | expected currently is my open question here -- i need to understand the plans/status for fixing. | 16:56 |
| kata-irc-bot | <sebastien.boeuf> this way, the daemon would process subsequent calls to set_mem_table as updates | 16:56 |
| eernst | again without, this has severe limitations | 16:56 |
| davidgiluk | yes agreed | 16:57 |
| davidgiluk | eernst: Can you tell me how kata hot plugs - specifically at what point? | 16:57 |
| eernst | I'll need a few, sure. | 16:58 |
| kata-irc-bot | <sebastien.boeuf> davidgiluk: stefanha: about the potential race condition, I think you can definitely put the daemon on hold for processing the queues while the set_mem_table is being processed. Is that what you have in mind with the new API (RCU like) Stefan? | 16:59 |
| *** jodh has quit IRC | 17:01 | |
| davidgiluk | seb: The tricky bit is that one of the threads might be blocked in a read() or something | 17:01 |
| stefanha | davidgiluk: It's a two-phase wait. First the vhost-user thread has to wait until all virtqueue processing threads have quiesced. Then the virtqueue processing threads have to wait until the vhost-user thread is done making changes. | 17:02 |
| stefanha | We should probably do this for other vhost-user messages that reconfigure things too. | 17:02 |
| stefanha | (and in the normal case no waiting happens because the vhost-user thread is mostly idle) | 17:03 |
| davidgiluk | stefanha: It's not always necessary; in the case where strictly only a new region is added and nothing else changes we're OK | 17:03 |
| stefanha | davidgiluk: Even then there is a race condition: what if the virtqueue thread runs before the hotplug event has been processed - it will think the memory region doesn't exist. | 17:04 |
| stefanha | Although I guess there is no way of preventing that in virtiofsd | 17:04 |
| davidgiluk | stefanha: The hotplug event sends an 'ack' back to the qemu, so the qemu is supposed to wait for the d to complete it | 17:04 |
| stefanha | It's up to QEMU and the guest to wait until hotplug is complete, so nevermind. | 17:04 |
| stefanha | Yep | 17:05 |
| davidgiluk | stefanha: There are some non obvious other cases where set-mem-table happens, changes to physical memory layout by dumb things like VGA reconfig | 17:14 |
| *** igordc has joined #kata-dev | 17:22 | |
| *** lpetrut has quit IRC | 17:30 | |
| *** davidgiluk has quit IRC | 19:23 | |
| *** pcaruana has quit IRC | 19:39 | |
| *** eernst has quit IRC | 19:51 | |
| kata-irc-bot | <jose.carlos.venegas.m> hey there what is the equivalent of journalctl -t kata-runtime for shimv2 in containerd ? | 19:53 |
| kata-irc-bot | <jose.carlos.venegas.m> @gmmaharaj @eric.ernst @salvador.fuentes ? | 19:54 |
| kata-irc-bot | <salvador.fuentes> I think it is only kata | 19:55 |
| kata-irc-bot | <salvador.fuentes> `journalctl -t kata` | 19:55 |
| kata-irc-bot | <jose.carlos.venegas.m> @salvador.fuentes right! thx | 19:57 |
| *** sameo has quit IRC | 20:43 | |
| *** fuentess has quit IRC | 21:04 | |
| *** igordc has quit IRC | 22:45 | |
| *** igordc has joined #kata-dev | 23:13 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!