| *** lcastell has quit IRC | 00:27 | |
| *** lcastell has joined #kata-general | 00:31 | |
| *** lcastell has quit IRC | 00:35 | |
| *** openstack has joined #kata-general | 02:26 | |
| *** ChanServ sets mode: +o openstack | 02:26 | |
| kata-irc-bot | <kmacleod> Hi folks. I’ve been trying to isolate a problem I’m seeing with Kata 1.5.x, I wanted to see if it rings any bells with anyone before I spend additional effort in trying to isolate it. We have a kube podspec with multiple containers. Some of those containers have CPU+memory limits, some do not. We’re finding that the containers which do not have limits, and which run a Java process in their entrypoint, are showing the JVM hangi | 03:22 |
|---|---|---|
| kata-irc-bot | immediately after startup. The JVMs look to be hanging on a kernel lock of some sort. The interesting part is that when we add a CPU limit to the affected containers, the problem goes away. Similarly, when we remove the CPU limits from all of the containers, the problem goes away. | 03:22 |
| kata-irc-bot | <eric.ernst> Very interesting. | 04:12 |
| kata-irc-bot | <eric.ernst> Is there a pod spec I can test with? | 04:12 |
| kata-irc-bot | <eric.ernst> @kmacleod | 04:12 |
| kata-irc-bot | <kmacleod> not yet, no, it’s all tied up with secrets and other peripheral stuff, so it’s hard to decouple into something standalone | 04:13 |
| kata-irc-bot | <kmacleod> if I come up with anything, I’ll post it here | 04:13 |
| kata-irc-bot | <kmacleod> given that there’s a workaround (to add cpu limits to every container), Im not sure I’ll get much time to spend on it | 04:14 |
| kata-irc-bot | <eric.ernst> Yeah. Can you control memory settings for the JVM? | 04:15 |
| kata-irc-bot | <eric.ernst> Or try with a larger default memory for kata as a test? | 04:15 |
| kata-irc-bot | <eric.ernst> I’m guessing JVM is eating more than it’s share | 04:16 |
| kata-irc-bot | <eric.ernst> And it’s impacting guest in general | 04:16 |
| kata-irc-bot | <eric.ernst> Understood. I’ll try to make something synthetic, but I’m guessing oom or something along these lines in the guest. | 04:17 |
| kata-irc-bot | <eric.ernst> Constraining the JVM container would protect you from this. | 04:18 |
| kata-irc-bot | <eric.ernst> Running with runc and seeing memory stats for the container would be interesting too | 04:18 |
| kata-irc-bot | <eric.ernst> Or just checking container level stats in general w kata | 04:18 |
| kata-irc-bot | <kmacleod> we don’t have an issue with memory limits, that I can tell, just cu limits | 04:20 |
| *** irclogbot_0 has quit IRC | 06:09 | |
| *** irclogbot_0 has joined #kata-general | 06:10 | |
| *** sgarzare has joined #kata-general | 08:16 | |
| *** sameo has joined #kata-general | 08:21 | |
| *** tmhoang has joined #kata-general | 08:32 | |
| *** gwhaley has joined #kata-general | 08:53 | |
| kata-irc-bot | <scethiren> hi may i do start and stop vm programatically ?? I want to do vm and container scaling in keta. | 10:07 |
| *** lpetrut has joined #kata-general | 10:33 | |
| *** gwhaley has quit IRC | 12:06 | |
| *** gwhaley has joined #kata-general | 12:59 | |
| *** sgarzare has quit IRC | 13:25 | |
| *** sgarzare has joined #kata-general | 13:28 | |
| *** openstack has joined #kata-general | 15:41 | |
| *** ChanServ sets mode: +o openstack | 15:41 | |
| *** igordc has quit IRC | 16:48 | |
| *** sameo has quit IRC | 16:49 | |
| *** igordc has joined #kata-general | 16:52 | |
| *** igordc has quit IRC | 16:53 | |
| *** igordc has joined #kata-general | 16:56 | |
| *** igordc has quit IRC | 17:10 | |
| kata-irc-bot | <david532> Before I go down the documentation rabbit hole today - in evaluating Kata for my own usage, I rely heavily on FUSE for my containers. In the limitations doc, I do not see anything about passing in devices or CAP adds. Am I correct to assume that `/dev/fuse` could be mounted from Kata to the container? | 17:14 |
| kata-irc-bot | <graham.whaley> Ooh, @david532 - that is not something I think I've seen anybody do, specifically fuse that is. the CAP adds should work fine. /cc @archana.m.shinde for any insights into adding /dev/fuse as a device.... Question then - what do you do with that /dev/fuse - where do you run the userland (in the container I guess), and which FS do you expect it to then have access to (the host?). I'm trying to think if there is going to be any | 17:17 |
| kata-irc-bot | potential clash between the fuse in the VM kernel (which may or may not be enabled - I don't know off the top of my head), or not... and you might need FUSE stuff enabled in the VM kernel I guess. | 17:17 |
| kata-irc-bot | <david532> @graham.whaley The use case here is mounting cloud storage (Google Drive, Dropbox, Google Cloud Storage) providers as a remote file system utilizing RClone, which in mount mode depends on FUSE. | 17:19 |
| kata-irc-bot | <david532> I wasn't sure if that was something already included in the stock kernel that ships with the standard package or if it would be something I would have to rebuild myself and append it in. | 17:19 |
| kata-irc-bot | <david532> I'm currently spinning up a Packet.net spot instance to retest this initial path because I wasn't comprehensive enough in my quick 30 minute test yesterday. | 17:20 |
| kata-irc-bot | <graham.whaley> this is our default kernel config for the VM's - and, looks to me like FUSE is off by default right now: https://github.com/kata-containers/packaging/blob/master/kernel/configs/x86_64_kata_kvm_4.19.x#L2600 | 17:21 |
| kata-irc-bot | <graham.whaley> so, to check this out you might need to build yourself a modified kernel. the tool is pretty easy to use: https://github.com/kata-containers/osbuilder | 17:21 |
| *** igordc has joined #kata-general | 17:21 | |
| kata-irc-bot | <graham.whaley> if you are OK doing kernel configs to set up your .config file... | 17:21 |
| kata-irc-bot | <graham.whaley> I think it is an interesting use case btw. /cc @manohar.r.castelino as well | 17:22 |
| kata-irc-bot | <david532> I've only built a kernel from source once, but it's worth a shot to give it another try. | 17:22 |
| kata-irc-bot | <graham.whaley> the osbuilder will do the kernel download/build. You can do it once, and then run 'make menuconfig' in that tree it makes to turn on FUSE, and then run the tool again to pick up the changes (or thereabouts). Hopefully the osbuilder docs cover that (make your own kernel) already. If not, ping @jose.carlos.venegas.m (or just here on this channel), depending on who is around/awake ;) | 17:24 |
| kata-irc-bot | <david532> Thanks. Let me try that today. Appreciate the quick responses. | 17:24 |
| kata-irc-bot | <david532> How are you guys running integration tests and such with something as virtualization dependent? | 17:25 |
| kata-irc-bot | <david532> Is it easiest to create a VM with nested virt and run Kata in that on a supported guest OS for "tests" | 17:25 |
| kata-irc-bot | <graham.whaley> our jenkins CI deploys cloud instances on Azure or vexxhost that support nested VMs. | 17:25 |
| kata-irc-bot | <graham.whaley> the 'metrics' and some other arch (arm, ibm power and s390) are using bare metal instances. the x86 metrics is using a packet one ;) | 17:26 |
| kata-irc-bot | <jose.carlos.venegas.m> @graham.whaley @david532 actually the kernel build bits are here | 17:27 |
| kata-irc-bot | <jose.carlos.venegas.m> https://github.com/kata-containers/packaging/tree/master/kernel | 17:27 |
| kata-irc-bot | <david532> I don't see any build OS/requirements; I assume it can be built anywhere that has Go/build tools (Does that include Docker - I don't explicitly see it where the guest OS has information)? | 17:30 |
| kata-irc-bot | <graham.whaley> @david532 - looks like I was wrong, and osbuilder only does the rootfs image, and the kernel build is over where @jose.carlos.venegas.m pointed. | 17:32 |
| kata-irc-bot | <graham.whaley> the osbuilder supports buiding inside a docker container to save the user having to set up their host distro (I think it works by default on fedora maybe as a native build). But, I'm not sure if the kernel scripts have a similar docker method or not ... over to @jose.carlos.venegas.m;-) | 17:33 |
| kata-irc-bot | <jose.carlos.venegas.m> let us know if you have any issue with it | 17:33 |
| kata-irc-bot | <jose.carlos.venegas.m> @graham.whaley they dont, but would be good to have it for that reason and have more reproducible builds | 17:34 |
| *** sgarzare has quit IRC | 17:34 | |
| *** igordc has quit IRC | 17:43 | |
| *** igordc has joined #kata-general | 17:48 | |
| kata-irc-bot | <david532> Unrelated but curiosity - what are the use cases for rebuilding the rootfs with the osbuilder? | 17:55 |
| *** tmhoang has quit IRC | 17:58 | |
| kata-irc-bot | <graham.whaley> @david532 - mostly we use those scripts in our CI system ;) and for development, but, there are valid end user use cases as well. We had one last week iirc to do with adding an extra systemd item to their VM startup (I forget exactly what though). | 18:08 |
| kata-irc-bot | <greg.bock> adding driver support as a kernel module, adding a debug shell to the console | 18:08 |
| kata-irc-bot | <greg.bock> to name a couple | 18:08 |
| kata-irc-bot | <graham.whaley> It also allows users to more easily set up for their own distro as the base VM OS (some users have that requirement), or add some module loads or plugins etc. etc. | 18:08 |
| kata-irc-bot | <david532> What are factories in relation to Kata? I wasn't able to find specific documentation explaining that as well. | 18:43 |
| kata-irc-bot | <graham.whaley> @david532 - you're right, I can't see any docs either. I thought I'd seen a PR just this last week adding some. @archana.m.shinde, did I maybe dream that? Anyway, the factories are, iirc, a way to cache warm VMs to improve startup time - and iirc also density. @eric.ernst @sebastien.boeuf - is that right. And, please pull in the factory author (let me see if I can find the right contact...) | 19:01 |
| kata-irc-bot | <david532> More info would be great on that - in my current use case I'm not too concerned about boot times on VMs, but it was one thing poking around that wasn't explained well for me. | 19:02 |
| kata-irc-bot | <graham.whaley> @david532 - here is the original code addition, that has some details. /cc @bergwolf :slightly_smiling_face: https://github.com/kata-containers/runtime/pull/303 | 19:02 |
| kata-irc-bot | <david532> Perfect! Thank you @graham.whaley | 19:02 |
| kata-irc-bot | <david532> You guys were at Kubecon Seattle last year, right? | 19:03 |
| kata-irc-bot | <david532> I swear that's how I originally heard of you. | 19:03 |
| kata-irc-bot | <david532> I could be dreaming, but I thought I remember getting a postcard sized ad. | 19:03 |
| kata-irc-bot | <graham.whaley> Oh, I reckon @eric.ernst was probably there, and some others... I wasn't (I'm 1/3rd of the way around the world, and they are close ;) ) | 19:04 |
| kata-irc-bot | <graham.whaley> yeah, I've seen those handout cards as well for kata - and handed some out in London.. | 19:04 |
| kata-irc-bot | <archana.m.shinde> @david532 Take a look at https://github.com/kata-containers/documentation/blob/master/how-to/what-is-vm-templating-and-how-do-I-use-it.md | 19:04 |
| kata-irc-bot | <graham.whaley> thx @archana.m.shinde - hah, that is what we get for having more than one phrase for the same thing (cache, vm factory, etc.) | 19:05 |
| kata-irc-bot | <david532> Thanks @archana.m.shinde I think that's why - was Google searching `kata containers factory` and only pulling code up; just a different name on the docs. | 19:05 |
| kata-irc-bot | <archana.m.shinde> There is also this https://github.com/kata-containers/documentation/blob/master/how-to/what-is-vm-cache-and-how-do-I-use-it.md | 19:05 |
| kata-irc-bot | <archana.m.shinde> describing vm-caches | 19:05 |
| kata-irc-bot | <graham.whaley> hmm, are they indexed from the top level docs repo toc ... if not, they probably should be. | 19:06 |
| kata-irc-bot | <archana.m.shinde> gwhaley yeah I dont think they are indexed | 19:06 |
| kata-irc-bot | <david532> `https://www.google.com/search?q=kata+containers+factory&oq=kata+containers+factory&aqs=chrome.0.69i59j69i60l3.1815j0j7&sourceid=chrome&ie=UTF-8` This is the specific query I was running. | 19:06 |
| kata-irc-bot | <david532> With the top three being specific reference to code. | 19:07 |
| kata-irc-bot | <archana.m.shinde> I remember we had this whole discussion about md vs rst | 19:07 |
| kata-irc-bot | <david532> And a "supports factory" for guest OS. | 19:07 |
| kata-irc-bot | <graham.whaley> @david532 - I was doing a markdown file search in the kata org on github, and was not finding them :slightly_smiling_face: | 19:07 |
| kata-irc-bot | Action: graham.whaley watches archana.m.shinde start the holy war of doc formats ... | 19:07 |
| kata-irc-bot | <david532> I didn't have the super luxury of time to dig deep since I was still running my Packet instance. | 19:07 |
| kata-irc-bot | <archana.m.shinde> haha @graham.whaley..not getting there ! | 19:08 |
| kata-irc-bot | <david532> And those $0.02/hr add up quiickly. | 19:08 |
| *** gwhaley has quit IRC | 19:27 | |
| *** igordc has quit IRC | 19:32 | |
| *** igordc has joined #kata-general | 19:33 | |
| kata-irc-bot | <david532> So once I've got this built,, I ran the install command on my build VM and see it's dropped: ``` INFO: Clone config and patches lrwxrwxrwx 1 root root 18 Mar 12 12:45 /usr/share/kata-containers/vmlinux.container -> vmlinux-4.19.24-30 lrwxrwxrwx 1 root root 18 Mar 12 12:45 /usr/share/kata-containers/vmlinuz.container -> vmlinuz-4.19.24-30 ``` and in the kata-containers usr share: ``` -rw-r--r-- 1 root root 81861 Mar 12 12:45 | 19:46 |
| kata-irc-bot | config-4.19.24 -rw-r--r-- 1 root root 28840496 Mar 12 12:45 vmlinux-4.19.24-30 lrwxrwxrwx 1 root root 18 Mar 12 12:45 vmlinux.container -> vmlinux-4.19.24-30 -rw-r--r-- 1 root root 6623280 Mar 12 12:45 vmlinuz-4.19.24-30 lrwxrwxrwx 1 root root 18 Mar 12 12:45 vmlinuz.container -> vmlinuz-4.19.24-30 ``` Can just slap these into the specified folder on my intended running host in the formats it has above? | 19:46 |
| *** igordc has quit IRC | 20:13 | |
| kata-irc-bot | <david532> I just ended up building it on the host itself - testing now. | 21:09 |
| kata-irc-bot | <david532> Shucks, I missed `support-for-joining-an-existing-vm-network` in limitations. | 21:22 |
| *** igordc has joined #kata-general | 21:26 | |
| *** igordc has quit IRC | 21:26 | |
| *** igordc has joined #kata-general | 21:27 | |
| *** igordc has quit IRC | 22:25 | |
| *** igordc has joined #kata-general | 22:43 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!