| *** ailan__ has joined #kata-dev | 07:04 | |
| *** pcaruana has joined #kata-dev | 07:14 | |
| *** davidgiluk has joined #kata-dev | 08:03 | |
| *** ailan__ has quit IRC | 08:17 | |
| *** ailan__ has joined #kata-dev | 08:17 | |
| *** jodh has joined #kata-dev | 08:21 | |
| *** ailan__ has quit IRC | 08:38 | |
| kata-irc-bot | <james.o.hunt> Any rustaceans out there who could help review https://github.com/kata-containers/kata-containers/pull/223 and https://github.com/kata-containers/kata-containers/pull/225? | 09:01 |
|---|---|---|
| *** devimc has joined #kata-dev | 12:07 | |
| *** devimc has quit IRC | 13:19 | |
| *** devimc has joined #kata-dev | 13:19 | |
| *** crobinso has joined #kata-dev | 13:41 | |
| *** pcaruana has quit IRC | 13:44 | |
| kata-irc-bot | <simon.kaegi> ah ok... thanks @eric.ernst just figured out that part is sort of mandatory ;) I'm going to see what happens if I merge the changes https://gitlab.com/virtio-fs/linux/-/commits/virtio-fs-dev/ into a recent 5.6. I noticed that at the overlayfs-next stuff seems to be merged already. Is there "any way" to support virtiofs without the dax changes? | 13:45 |
| *** pcaruana has joined #kata-dev | 13:45 | |
| kata-irc-bot | <simon.kaegi> At a glance it looked like the overlayfs stuff was merged in but there is still a bunch of DAX stuff that is not. e.g. all the changes from 9 Apr 2020 here -- https://gitlab.com/virtio-fs/linux/-/commits/virtio-fs-dev/ apart from the top commit. I'm out of my league here but am looking at what merging those changes on current 5.6 will look like. | 13:48 |
| *** pcaruana has quit IRC | 13:49 | |
| *** pcaruana has joined #kata-dev | 13:50 | |
| kata-irc-bot | <fidencio> Hey! We do support virtiofs without the dax changes | 14:02 |
| kata-irc-bot | <fidencio> Do you still have the config I shared with you? | 14:02 |
| kata-irc-bot | <fidencio> If you set `virtio_fs_cache_size = 0` dax won't be used and you're fine and good to go | 14:03 |
| *** dklyle has joined #kata-dev | 14:14 | |
| kata-irc-bot | <eric.ernst> Yes. | 14:16 |
| kata-irc-bot | <eric.ernst> Performance just suffers, that’s all. | 14:16 |
| kata-irc-bot | <eric.ernst> To clarify; what part is mandatory? | 14:16 |
| *** pcaruana has quit IRC | 14:49 | |
| *** pcaruana has joined #kata-dev | 14:49 | |
| kata-irc-bot | <simon.kaegi> ok awesome thanks!! will try | 14:49 |
| kata-irc-bot | <simon.kaegi> Also looks like the overlayfs stuff is only in 5.7 at least at first glance | 14:50 |
| kata-irc-bot | <fidencio> I can double check with Vivek, but it seems to be merged on 5.6 | 14:53 |
| kata-irc-bot | <simon.kaegi> thanks, I'm just looking at the diffs and it very well might be as I don't understand the core of what was added | 14:56 |
| *** jugs1 has quit IRC | 15:20 | |
| *** jugs1 has joined #kata-dev | 15:28 | |
| kata-irc-bot | <simon.kaegi> ... and success -- thanks | 15:35 |
| kata-irc-bot | <simon.kaegi> Let me try the overlay support | 15:35 |
| davidgiluk | eric.ernst: DAX is not mandatory | 15:36 |
| kata-irc-bot | <fidencio> Cool! If something goes wrong, let us know, | 15:36 |
| kata-irc-bot | <simon.kaegi> perf for my test was a little slower than with dax but still much much better than for 9p | 15:36 |
| kata-irc-bot | <simon.kaegi> ... and you're totally right -- overlayfs work in 5.6.15 | 15:40 |
| kata-irc-bot | <simon.kaegi> woot! | 15:40 |
| kata-irc-bot | <simon.kaegi> I am "super" happy with this | 15:40 |
| kata-irc-bot | <fidencio> \o/ | 15:42 |
| kata-irc-bot | <simon.kaegi> So... I obviously have a ton of testing to do now... but this is starting to look really good. | 15:44 |
| kata-irc-bot | <simon.kaegi> I'm still using the virtiofs-dev qemu so next step is to switch over to 5.0 | 15:44 |
| *** sameo has joined #kata-dev | 16:42 | |
| *** dklyle has quit IRC | 17:23 | |
| *** dklyle has joined #kata-dev | 17:23 | |
| kata-irc-bot | <simon.kaegi> It turns out none of it is. I just needed to set virtio_fs_cache_size = 0. In terms of perf loss in my simple `dd` test on one of our managed bare metal kubernetes nodes I'm getting: 1. default (runc) -- 185 MB/s --- [100%] 2. kata-qemu -- 105 MB/s --- [56%] 3. kata-qemu-virtiofs -- 165 MB/s --- [89%] 4. kata-clh -- 175 MB/s --- [95%] 5. kata-qemu-virtiofs-nodax -- 157 MB/s --- [85%] | 17:42 |
| *** sunnycai has joined #kata-dev | 17:46 | |
| *** jodh has quit IRC | 18:00 | |
| devimc | @simon.kaegi are you interested on IO performance? | 18:20 |
| kata-irc-bot | <simon.kaegi> definitely but currently we're using 9p so even no dax virtiofs is a giant improvement | 18:21 |
| devimc | might be you want to take a look to this https://github.com/intel/pmem-csi | 18:21 |
| devimc | kata already supports pmem devices | 18:21 |
| davidgiluk | simon.kaegi: The DAX trick works better for some stuff than others; it's bad for huge sequentials - it's much better when you have a reasonably small working set - one that fits in the cache size you set | 18:41 |
| fidencio | devimc: do you recall some issue with OpenShift and getting the stats from the kata pods? or whether it should be working with shimv2? | 19:07 |
| devimc | fidencio: stats? like cadvisor stats? | 19:08 |
| devimc | fidencio: I guess you are facing issues with IO and network stats | 19:10 |
| fidencio | devimc: like cadvisor stats, but AFAIU cadvisor is not supposed to work and cri-stats should be the ones used | 19:10 |
| fidencio | devimc: yep | 19:11 |
| devimc | fidencio: I guess you are using 9p .. ? | 19:11 |
| fidencio | devimc: never! | 19:11 |
| devimc | mmm | 19:11 |
| devimc | is sandbox_cgroup_only true? | 19:12 |
| fidencio | devimc: I know that if k8s get crio.sock as the runtime_endpoint, it'll fallback do cadvisor .. and I've changed that on both crio.conf and kubelet | 19:12 |
| fidencio | sandbox_cgroup_only=true, yes | 19:13 |
| devimc | fidencio: take a look to this https://github.com/kata-containers/runtime/issues/2400 | 19:16 |
| *** crobinso has quit IRC | 19:16 | |
| fidencio | devimc: hmmm. seems to be related but still reproducible with virtiofs | 19:21 |
| *** davidgiluk has quit IRC | 19:23 | |
| fidencio | devimc: if I'm going to report that upstream to keep track of what we're doing, would you prefer to have that to extend 2400 or as new issue? | 19:54 |
| devimc | fidencio: feel free to raise a new issue, we can close it later if it's duplicated | 19:55 |
| fidencio | devimc: okkie | 19:55 |
| *** sameo has quit IRC | 20:45 | |
| *** sameo has joined #kata-dev | 20:47 | |
| *** th0din has quit IRC | 20:51 | |
| *** th0din has joined #kata-dev | 20:52 | |
| *** th0din has quit IRC | 20:54 | |
| *** devimc has quit IRC | 20:56 | |
| *** th0din has joined #kata-dev | 21:18 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!