| *** crobinso has quit IRC | 01:06 | |
| *** sameo has joined #kata-dev | 06:19 | |
| *** dklyle has quit IRC | 07:57 | |
| *** sgarzare has joined #kata-dev | 07:57 | |
| *** fgiudici has joined #kata-dev | 08:12 | |
| *** jodh has joined #kata-dev | 08:27 | |
| *** fgiudici_ has joined #kata-dev | 08:51 | |
| *** fgiudici has quit IRC | 08:51 | |
| *** davidgiluk has joined #kata-dev | 09:01 | |
| *** fgiudici_ has quit IRC | 09:04 | |
| *** fgiudici has joined #kata-dev | 09:04 | |
| *** fgiudici has quit IRC | 09:25 | |
| *** fgiudici has joined #kata-dev | 09:25 | |
| *** ailan has joined #kata-dev | 09:33 | |
| *** devimc has joined #kata-dev | 13:05 | |
| *** crobinso has joined #kata-dev | 13:37 | |
| *** fuentess has joined #kata-dev | 14:04 | |
| *** jodh has quit IRC | 14:11 | |
| *** jodh has joined #kata-dev | 14:20 | |
| devimc | Hi davidgiluk, | 14:53 |
|---|---|---|
| devimc | qq, how can I debug virtiofs ? | 14:53 |
| davidgiluk | devimc: Ah, well, how did you break it? | 14:54 |
| devimc | davidgiluk, running cloud-hypervisor + kata + knative | 14:54 |
| devimc | any operation on the shared directory is getting stuck | 14:55 |
| davidgiluk | that's an exciting combination - are you running the C daemon or the rust daemon? | 14:55 |
| devimc | davidgiluk, for example ls /run/kata-containers/shared/containers | 14:55 |
| devimc | is getting stuck | 14:55 |
| devimc | davidgiluk, C daemon | 14:55 |
| davidgiluk | devimc: Stuck normally means the daemon has crashed, so if you have a core or something then that would be great, if not then you can try running the daemon with -o debug and see what it's dieing breath is | 14:56 |
| devimc | davidgiluk, thx, let me enable core file dumps | 14:57 |
| davidgiluk | devimc: With all the sandboxing, finding where and if core dumps actually land can be tricky | 14:58 |
| devimc | davidgiluk, ok, I will use strace then | 14:59 |
| davidgiluk | devimc: That's a bit desperate; before doing that, I'd check /var/log/audit to see if there are any seccomp warns if you've tripped across a missing syscall | 15:01 |
| devimc | davidgiluk, type=ANOM_ABEND msg=audit(1611606817.326:32314): auid=4294967295 uid=0 gid=0 ses=4294967295 pid=38414 comm="pool" exe="/opt/kata/bin/v | 15:05 |
| devimc | irtiofsd" sig=11 res=1 | 15:05 |
| devimc | I guess sig is signal | 15:05 |
| davidgiluk | sniff, a segfault | 15:06 |
| devimc | :( | 15:07 |
| davidgiluk | devimc: Then a core or gdb is probably best, although a -o debug might make it obvious | 15:07 |
| devimc | davidgiluk, really hard to find the log, since this is a scalability issue | 15:07 |
| davidgiluk | devimc: Can you get a gdb to it before it crashes? | 15:07 |
| devimc | davidgiluk, sure, I will enable core file dump | 15:08 |
| kata-irc-bot | <eric.ernst> devimc, is this with dax? | 15:15 |
| *** dklyle has joined #kata-dev | 15:16 | |
| devimc | davidgiluk, I think DAX is disabled | 15:17 |
| devimc | virtio_fs_cache_size = 0 | 15:17 |
| davidgiluk | yeh | 15:18 |
| kata-irc-bot | <eric.ernst> How often/quickly does this manifest? | 15:19 |
| devimc | all the time when running iostio + knative | 15:20 |
| devimc | *istio | 15:21 |
| kata-irc-bot | <eric.ernst> Having an issue is bad, but at least its easy to reproduce. | 15:21 |
| davidgiluk | nod, repeatable crashes are lovely | 15:21 |
| davidgiluk | devimc: Does it crash immediately on connect or later? | 15:22 |
| kata-irc-bot | <eric.ernst> I'm glad you're seeing this with CLH as well; its a bit easier for initial logs/debug. Can you repro same setup with qemu? | 15:22 |
| devimc | davidgiluk, latter, when envoy runs | 15:22 |
| devimc | envoy is the second container | 15:22 |
| devimc | each pod has 2 containers | 15:23 |
| davidgiluk | devimc: OK, so glue gdb to it, and then start your 2nd container | 15:26 |
| kata-irc-bot | <gabriela.cervantes.te> @eric.ernst yes, I am working on a PR to bring the latest changes to stable-2.0 for the CI | 15:43 |
| devimc | davidgiluk, https://paste.centos.org/view/a94eb350 | 15:58 |
| devimc | coredump is not really useful, I need to enable debug and rebuild it | 15:58 |
| davidgiluk | devimc: I'm not sure you've traced the interesting threads there | 16:15 |
| devimc | davidgiluk, let me try again. now with virtiofsd symbols | 16:19 |
| devimc | *debug symbols | 16:19 |
| *** pmores has joined #kata-dev | 16:33 | |
| kata-irc-bot | <fidencio> @samuel.ortiz, out of curiosity, we talked about devicemapper on the meeting, but what's the devicemapper advantages over overlayfs? Or, maybe a better question, do you see performance impacts on someone using overlayfs instead of devicemapper? If so, why? | 16:51 |
| pmores | Hi! I'm wondering what keeps https://github.com/kata-containers/runtime/pull/3127 from being merged - it has approvals but several checks, marked as Required, seem to be Expected. | 17:01 |
| pmores | Considering that last changes to the PR seem to be almost two weeks old, is this OK? | 17:02 |
| kata-irc-bot | <eric.ernst> @fidencio -- just do an fio test using blk devcie and using virtiofs | 17:05 |
| kata-irc-bot | <eric.ernst> perf wise, virtio-blk > virtiofs (today) | 17:05 |
| *** devimc has quit IRC | 17:20 | |
| *** devimc has joined #kata-dev | 17:21 | |
| kata-irc-bot | <fidencio> We have to run /test there, Pavel. | 17:23 |
| pmores | fidencio: thanks! | 17:25 |
| *** fgiudici has quit IRC | 18:00 | |
| *** jodh has quit IRC | 18:00 | |
| *** ailan has quit IRC | 18:06 | |
| *** sgarzare has quit IRC | 18:23 | |
| kata-irc-bot | <fidencio> @eric.ernst, @jose.carlos.venegas.m, do we have someone from firecracker community that we can loop into our issues? | 18:47 |
| kata-irc-bot | <eric.ernst> They've been pretty responsive / collaborative. | 19:06 |
| kata-irc-bot | <eric.ernst> Probably depends on the area of the issue. | 19:06 |
| kata-irc-bot | <fidencio> I've opened https://github.com/kata-containers/tests/issues/3268 and marked Penny there. | 19:18 |
| kata-irc-bot | <fidencio> If you think there's someone else that could help, please, mark the person | 19:19 |
| kata-irc-bot | <jose.carlos.venegas.m> @fidencio I may be wrong but I think @penny.zheng is not doing a lot of kata recently | 19:35 |
| kata-irc-bot | <fidencio> Hmm. Let me open an issue on Firecracker then | 19:35 |
| kata-irc-bot | <fidencio> And ask for help on their community (as I did for acrn) | 19:35 |
| kata-irc-bot | <fidencio> Thanks, @jose.carlos.venegas.m! | 19:35 |
| kata-irc-bot | <jose.carlos.venegas.m> @fidencio, no problem, lets see if someone has some cycles there, else ask if someone want to take ownership of the integration in kata | 19:36 |
| kata-irc-bot | <jose.carlos.venegas.m> it may be good to get in good shape the integration for the usecases that makesense for fc | 19:37 |
| kata-irc-bot | <jose.carlos.venegas.m> mostly kata + fass would be the main use case | 19:37 |
| kata-irc-bot | <fidencio> @gabriela.cervantes.te, @jose.carlos.venegas.m, let's talk a little bit about CI :slightly_smiling_face: :thread: | 19:42 |
| kata-irc-bot | <fidencio> I've noticed that the majority of our integration tests, if not all of them, depend on docker :slightly_smiling_face: | 19:42 |
| kata-irc-bot | <fidencio> I'm talking about `main` (so, 2.x) of the tests repo | 19:42 |
| kata-irc-bot | <fidencio> Can we just remove all of those? | 19:42 |
| kata-irc-bot | <jose.carlos.venegas.m> all jobs depending in docker? | 19:43 |
| kata-irc-bot | <fidencio> 1 minute, let me get links. | 19:43 |
| kata-irc-bot | <jose.carlos.venegas.m> I think most of the (it not all should depend as a way to build some assets) but we dont to any testing using docker | 19:43 |
| kata-irc-bot | <jose.carlos.venegas.m> I say because today kata 2.x is not compatible with any docker version | 19:44 |
| kata-irc-bot | <fidencio> We're not testing, but the tests are still there. Take a look here: https://github.com/kata-containers/tests/tree/main/integration | 19:44 |
| kata-irc-bot | <fidencio> If you do a `git grep docker.*runtime`, you get: https://paste.centos.org/view/17c7d2ce | 19:45 |
| kata-irc-bot | <fidencio> All of those could be simply removed | 19:45 |
| kata-irc-bot | <fidencio> And I already removed docker and docker-composed ones | 19:46 |
| kata-irc-bot | <fidencio> Here's the full list of things that can be removed: https://paste.centos.org/view/cc0baf7e | 19:46 |
| kata-irc-bot | <jose.carlos.venegas.m> I see, yep there is a lot of them that are not running today, I am not sure what should be the best way. | 19:47 |
| kata-irc-bot | <fidencio> Argh! https://paste.centos.org/view/90ba72ea | 19:47 |
| kata-irc-bot | <jose.carlos.venegas.m> we could left them until the time docker is back, or remove them and if docker is back re add them | 19:47 |
| kata-irc-bot | <jose.carlos.venegas.m> @fidencio for some of those we are updating them to use ctr intead | 19:48 |
| kata-irc-bot | <fidencio> My suggestion is: 1. Open an issue about removing everything that's docker related and not tested on 2.x; 2. Remove everything, no mercy; 3. Create issues for evaluating what should be added again; | 19:49 |
| kata-irc-bot | <fidencio> IMHO, removing something not used at all and waiting for people to raise a complain (thus step 3) seems to work better than ask to remove | 19:49 |
| kata-irc-bot | <fidencio> And git, thankfully, allows us to get to those files quite easily if needed. | 19:50 |
| kata-irc-bot | <fidencio> (for those young kids who never used anything else but git, you're spoiled! :-)) | 19:50 |
| kata-irc-bot | <gabriela.cervantes.te> we can create a PR to remove all docker tests and maybe people can comment or hate the PR | 19:50 |
| kata-irc-bot | <jose.carlos.venegas.m> yeah, it is not that they will be gone forever | 19:50 |
| kata-irc-bot | <fidencio> Ack, let me open issues and submit a PR then! | 19:50 |
| kata-irc-bot | <gabriela.cervantes.te> ok thanks | 19:51 |
| kata-irc-bot | <fidencio> Thanks a lot for the input! | 19:51 |
| *** crobinso has quit IRC | 20:10 | |
| kata-irc-bot | <fidencio> Actually, I opened the issue but I'll not do the change right now as I realised it may be a good first issue for someone looking at the project. | 20:10 |
| *** davidgiluk has quit IRC | 20:14 | |
| kata-irc-bot | <jose.carlos.venegas.m> :slightly_smiling_face: | 20:19 |
| kata-irc-bot | <jose.carlos.venegas.m> we need a away to announce good first issues | 20:19 |
| kata-irc-bot | <jose.carlos.venegas.m> a weekly mail asking for help or a slack message | 20:19 |
| kata-irc-bot | <jose.carlos.venegas.m> some youtube ads | 20:20 |
| kata-irc-bot | <fidencio> I'm trying to remember who of you people hooked me up into this project. | 20:24 |
| kata-irc-bot | <fidencio> I don't remember what was done, but somehow worked. :slightly_smiling_face: | 20:24 |
| *** pmores has quit IRC | 21:12 | |
| *** sameo has quit IRC | 22:08 | |
| *** devimc has quit IRC | 22:09 | |
| *** kgz has quit IRC | 22:46 | |
| *** fidencio has quit IRC | 22:46 | |
| *** kgz has joined #kata-dev | 22:47 | |
| *** fidencio has joined #kata-dev | 22:48 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!