| *** sameo has quit IRC | 01:07 | |
| *** sgarzare has joined #kata-general | 07:46 | |
| *** sameo has joined #kata-general | 09:06 | |
| *** devimc has joined #kata-general | 13:06 | |
| *** fuentess has joined #kata-general | 13:28 | |
| *** devimc has quit IRC | 14:59 | |
| *** devimc has joined #kata-general | 15:02 | |
| *** devimc has quit IRC | 15:37 | |
| *** devimc has joined #kata-general | 15:40 | |
| kata-irc-bot | <ieb> we think its related to thread creation. Perhaps a config setting. Over 1024 threads the time take to create another thread goes up dramatically. eg on a non kata container, the 1025th thread would take 0.4ms to create, but on kata it would take 17ms. By the 1501th thread that becomes 290ms. Its the number of threads running, not the number created since the process started. | 17:28 |
|---|---|---|
| kata-irc-bot | <ieb> Orange line is a runc container, blue line is kata on QEMU/KVM etc, same docker image used in both cases. | 17:30 |
| kata-irc-bot | <fidencio> @ieb, not that I'll be able to look at this right now, but I'd be interested (and it would help others to help you as well) to take a look at the kata-containers configuration file you have. Would be possible to share that one? | 17:30 |
| kata-irc-bot | <efoster> Let me do that | 17:31 |
| kata-irc-bot | <efoster> | 17:32 |
| kata-irc-bot | <efoster> Here's what the symlinks look like for our images: ```vmss-agent-kata0-14stageva7-qavvb00000D ~ # ls -l /opt/kata/share/kata-containers/ total 333756 -rw-r--r--. 1 root root 67335 Nov 12 07:19 config-5.4.32 -rw-r--r--. 1 root root 68443 Nov 12 07:19 config-kata-v5.6-april-09-2020 -rw-r--r--. 1 root root 268435456 Nov 12 07:22 kata-containers-image_clearlinux_1.11.5_agent_bdbea6619a.img lrwxrwxrwx. 1 root root 62 Nov 12 | 17:33 |
| kata-irc-bot | 07:22 kata-containers-initrd.img -> ./kata-containers-initrd_alpine_1.11.5_agent_bdbea6619a.initrd -rw-r--r--. 1 root root 13670454 Nov 12 07:13 kata-containers-initrd_alpine_1.11.5_agent_bdbea6619a.initrd lrwxrwxrwx. 1 root root 62 Nov 12 07:22 kata-containers.img -> ./kata-containers-image_clearlinux_1.11.5_agent_bdbea6619a.img -rw-r--r--. 1 root root 24751224 Nov 12 07:19 vmlinux-5.4.32-76 -rw-r--r--. 1 root root 24839424 Nov 12 | 17:33 |
| kata-irc-bot | 07:19 vmlinux-kata-v5.6-april-09-2020-76-virtiofs lrwxrwxrwx. 1 root root 43 Nov 12 07:19 vmlinux-virtiofs.container -> vmlinux-kata-v5.6-april-09-2020-76-virtiofs lrwxrwxrwx. 1 root root 17 Nov 12 07:19 vmlinux.container -> vmlinux-5.4.32-76 -rw-r--r--. 1 root root 4869920 Nov 12 07:19 vmlinuz-5.4.32-76 -rw-r--r--. 1 root root 4986096 Nov 12 07:19 vmlinuz-kata-v5.6-april-09-2020-76-virtiofs lrwxrwxrwx. 1 root root 43 | 17:33 |
| kata-irc-bot | Nov 12 07:19 vmlinuz-virtiofs.container -> vmlinuz-kata-v5.6-april-09-2020-76-virtiofs lrwxrwxrwx. 1 root root 17 Nov 12 07:19 vmlinuz.container -> vmlinuz-5.4.32-76``` | 17:33 |
| kata-irc-bot | <efoster> QEMU version: ```vmss-agent-kata0-14stageva7-qavvb00000D ~ # /opt/kata/bin/qemu-virtiofs-system-x86_64 -version QEMU emulator version 5.0.0 (kata-static) Copyright (c) 2003-2020 Fabrice Bellard and the QEMU Project developers``` | 17:33 |
| kata-irc-bot | <fidencio> ack! | 17:34 |
| kata-irc-bot | <fidencio> I am wondering here what would be the easiest way to try to *isolate* which component could be the one to blame for this. For instance, could we try to run a similar, if not the same test, just inside a VM (still using QEMU, still using virtiofs)? We can get the command line options from what we're using for Kata and launch the VM in a similar way. | 17:38 |
| kata-irc-bot | <efoster> Should be doable! I need the `qemu-virtiofs-system-x86_64` CLI args, right? | 17:40 |
| kata-irc-bot | <fidencio> right, you can easily get that if you can ssh into the node you're running the workload. | 17:42 |
| *** sgarzare has quit IRC | 17:43 | |
| kata-irc-bot | <efoster> | 17:46 |
| kata-irc-bot | <fidencio> @efoster, @ieb, also, I'd strongly recommend opening an issue for this. Then I can easily loop in the QEMU folks on that. | 17:48 |
| kata-irc-bot | <efoster> I'll get that written up today | 17:48 |
| kata-irc-bot | <fidencio> Thanks! | 17:59 |
| kata-irc-bot | <efoster> We're running 1.11.5, so should I file this issue on kata-containers/runtime? | 18:14 |
| kata-irc-bot | <efoster> Or would kata-containers/kata-containers be better for visibility? | 18:14 |
| *** devimc has quit IRC | 21:58 | |
| *** sameo has quit IRC | 23:11 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!