| *** igordc has quit IRC | 03:42 | |
| *** sameo has joined #kata-dev | 05:10 | |
| *** dklyle has quit IRC | 06:53 | |
| *** david-lyle has joined #kata-dev | 06:53 | |
| *** sgarzare has joined #kata-dev | 07:04 | |
| *** jodh has joined #kata-dev | 07:12 | |
| *** amorenoz has quit IRC | 07:36 | |
| *** amorenoz has joined #kata-dev | 07:36 | |
| *** ailan has joined #kata-dev | 07:51 | |
| *** davidgiluk has joined #kata-dev | 08:03 | |
| *** gwhaley has joined #kata-dev | 08:11 | |
| *** gwhaley has quit IRC | 08:15 | |
| *** gwhaley has joined #kata-dev | 08:15 | |
| *** gwhaley has quit IRC | 08:31 | |
| *** gwhaley has joined #kata-dev | 08:49 | |
| kata-irc-bot1 | <graham.whaley> @errordeveloper - I've not read all your details, but I will note fyi, that there is a good reason we have osbuilder with a set of known to work kernels. You will (as you have found) run into some challenges trying to use other kernels - starting with missing features by default or missing patches to fix subtle bugs when using VMs or nested VMs etc. And then, ultimately you will probably end up with a kernel that is larger and | 09:10 |
|---|---|---|
| kata-irc-bot1 | thus slower and has a bigger attack surface than those that come with Kata. That is fine as long as that is what you want/need/understand - and we can help you figure out what is missing/broken/misconfigured. But, ultimately our focus is on the kernels we ship with kata, you understand... | 09:10 |
| kata-irc-bot1 | <errordeveloper> @graham.whaley I see, that makes sense! | 09:12 |
| kata-irc-bot1 | <errordeveloper> > what is missing/broken/misconfigured. | 09:12 |
| kata-irc-bot1 | <errordeveloper> ok, so here is another one that tripped me up https://github.com/kata-containers/agent/issues/767 | 09:19 |
| *** ailan has quit IRC | 09:34 | |
| kata-irc-bot1 | <errordeveloper> I think I know how to get past the scsi scan error in agent code, I’ll do some testing and report back | 10:01 |
| *** ailan has joined #kata-dev | 10:19 | |
| kata-irc-bot1 | <errordeveloper> https://github.com/kata-containers/agent/pull/768 | 11:41 |
| kata-irc-bot1 | <errordeveloper> a review would be great, hope it’s all straightforward :-) | 11:41 |
| *** crobinso has joined #kata-dev | 12:05 | |
| kata-irc-bot1 | <graham.whaley> reviews normally come rolling in over a 24-48h period - developers spread across the globe and timezones :slightly_smiling_face: | 12:28 |
| davidgiluk | errordeveloper: ANy idea why it's erroring? | 12:59 |
| davidgiluk | errordeveloper: And what error do you get? | 13:00 |
| *** devimc has joined #kata-dev | 13:39 | |
| errordeveloper | davidgiluk: the scsi scan? I don't really know, but I think the code is written with certain assumption that don't work on the kernel I'm using | 13:53 |
| errordeveloper | davidgiluk: see the issue that the PR recences :) | 13:53 |
| davidgiluk | errordeveloper: Well so the question is whether you should ignore the error or is there something we should be doing so it actually works on both kernels | 13:54 |
| kata-irc-bot1 | Action: graham.whaley suspects ignoring the error loses us some function, even if that is not a function you are using, we cannot just ignore that for a kata kernel... I suspect it is volume or device plug or hotplug | 13:58 |
| errordeveloper | gwhaley: maybe, but my PR doesn't enable this by default, it could be seen as a debug sort of flag | 14:01 |
| errordeveloper | ok, so here is another thing - right now containers in kata VM see VMs kernel arguments, is there any way to mask those? | 14:03 |
| errordeveloper | I am running systemd inside my container, I don't want it to notice any systemd flags that kata uses, namely `systemd.unit=kata-containers.target systemd.mask=systemd-networkd.service systemd.mask=systemd-networkd.socket` | 14:04 |
| errordeveloper | and in debug mode that get some debug flags, I definetly see that as even bigger problem | 14:04 |
| kata-irc-bot1 | <graham.whaley> that's an interesting question. I think kata should be in the same position as a soft container - if using a soft container (runc), do you get to see the host systemd settings... if not, then there must be a namespace type way to mask them, and kata should be using that same mechanism inside the VM... | 14:11 |
| kata-irc-bot1 | <graham.whaley> the kata agent afaik uses the same libcontainer library as runc to do the isolation, so it should be the same already I would expect. | 14:11 |
| errordeveloper | gwhaley: ok, so in a docker container I can see /proc/cmdline from my host | 14:23 |
| errordeveloper | gwhaley: good to know that kata uses libcontainer internally | 14:23 |
| kata-irc-bot1 | <graham.whaley> there is an interesting RH article at https://developers.redhat.com/blog/2019/04/24/how-to-run-systemd-in-a-container/ that might have some clues - maybe podman helps/solves. Dan has been posting patches recently to kata :slightly_smiling_face: davidgiluk may also know some podman stuff.. | 14:24 |
| errordeveloper | gwhaley: ah, I don't think I've seen that one, I'll have a look | 14:24 |
| errordeveloper | I'm able to run systemd already, I'm just having a few concerns while poking around :) | 14:25 |
| errordeveloper | I'm glad to hear it just works on fedora in podman =) | 14:26 |
| kata-irc-bot1 | <graham.whaley> keep in mind, a kata container should look as much like a soft container as it can - if you can tell you are in a VM then maybe it is not doing its job. Sure, there is some stuff we cannot hide etc., but the idea is that 'containers just work', but happen to be wrapped in a VM | 14:26 |
| errordeveloper | 'Podman in systemd unit files works better than Docker | 14:26 |
| errordeveloper | =) | 14:26 |
| errordeveloper | no comment. | 14:26 |
| davidgiluk | gwhaley: I'll admit to not knowing much about it; but the question of kernel command line is interesting; I know Kata does mounts to hide bits of /proc and /sys, could we mount a dummy empty file over /proc/cmdline ? | 14:28 |
| kata-irc-bot1 | <graham.whaley> davidgiluk: well, if runc exposes the host cmdline, then one could argue kata should expose the vm commandline ;) - or, should be copy in and fake the host one - debateable.... | 14:29 |
| davidgiluk | gwhaley: It would seem better to fake one from config | 14:36 |
| errordeveloper | yeah, arguably it's a bug in "soft containers" and kata should do better! =) | 14:40 |
| kata-irc-bot1 | <graham.whaley> we try not to do 'better', as it tends to break existing containers... | 14:41 |
| errordeveloper | I wonder what may depend on such a "feature"! | 14:41 |
| kata-irc-bot1 | <graham.whaley> the 'better' would be to fix it in the upstream (moby/runc/libcontainer), but in this case, I don't think that is something that needs fixing there | 14:41 |
| kata-irc-bot1 | <graham.whaley> people write all sorts of broken containers. I even hear some folks put systemd in them :P | 14:42 |
| errordeveloper | right! | 14:43 |
| errordeveloper | well, my usecase is testing | 14:43 |
| davidgiluk | errordeveloper: The kernel where you're seeing the scsi rescan problem, is it one you built yourself or one from a distro? | 14:53 |
| errordeveloper | davidgiluk: it's one from linuxkit | 15:00 |
| davidgiluk | errordeveloper: So does the rescan file exist on it? And what error do you actually get? | 15:01 |
| errordeveloper | it said "invalid argument" | 15:02 |
| errordeveloper | davidgiluk: I've not been able to check if the file exists, I haven't managed to setup a debug shell | 15:03 |
| davidgiluk | seems odd doesn't it; if there was no hotplug configured in the kernel then I'd expect the file to be missing, so I wonder where the inval comes from | 15:04 |
| *** david-lyle has quit IRC | 15:08 | |
| *** dklyle has joined #kata-dev | 15:08 | |
| errordeveloper | I don't know, I can share what I have here, if you are keen to debug | 15:17 |
| errordeveloper | I have a managed to get a few things working now, so I'm just expiditing my poc =) | 15:18 |
| davidgiluk | errordeveloper: Not that keen :-) However, it seems wrong to just ignore an error if we don't understand why it happens | 15:25 |
| *** devimc has quit IRC | 15:30 | |
| *** devimc has joined #kata-dev | 15:30 | |
| errordeveloper | hm, so I am seeing an issue running containerd inside my kata pod | 15:44 |
| errordeveloper | root@test-cluter-master-55f47d5474-cw4rb:/# ctr run docker.io/library/hello-world:latest test1 | 15:44 |
| errordeveloper | ctr: io.containerd.runc.v2: failed to adjust OOM score for shim: set shim OOM score: write /proc/5447/oom_score_adj: invalid argument | 15:44 |
| errordeveloper | : exit status 1: unknown | 15:44 |
| errordeveloper | root@test-cluter-master-55f47d5474-cw4rb:/# systemctl status | 15:44 |
| errordeveloper | ● test-cluter-master-55f47d5474-cw4rb | 15:44 |
| errordeveloper | State: running | 15:44 |
| errordeveloper | Jobs: 0 queued | 15:44 |
| errordeveloper | Failed: 0 units | 15:44 |
| errordeveloper | Since: Thu 2020-04-02 15:09:33 UTC; 33min ago | 15:44 |
| errordeveloper | CGroup: /system.slice/kata-agent.service/kubepods-besteffort-pod12ab7de5_d105_4384_9e81_e909293957f4.slice:cri-containerd:bfd8b87227ebab3a33f0f72c879a0b8708f37be5d0c7226ca75b100f855a476f | 15:44 |
| errordeveloper | ├─init.scope | 15:44 |
| errordeveloper | │ └─1 /lib/systemd/systemd | 15:44 |
| errordeveloper | └─system.slice | 15:44 |
| errordeveloper | ├─containerd.service | 15:44 |
| errordeveloper | │ └─285 /usr/bin/containerd | 15:44 |
| errordeveloper | ├─systemd-journald.service | 15:44 |
| errordeveloper | │ └─22 /lib/systemd/systemd-journald | 15:45 |
| errordeveloper | ├─systemd-resolved.service | 15:45 |
| errordeveloper | │ └─280 /lib/system | 15:45 |
| errordeveloper | sorry - too many lines, haven't used IRC in a long time... | 15:45 |
| errordeveloper | ah, nevermind, seems like a systemd thing | 15:47 |
| errordeveloper | but I do wonder - what does my container inside kata VM get in terms of privileges? | 15:47 |
| gwhaley | its.... a container - it gets whatever privs you asked docker to give it - iyswim.... | 15:55 |
| gwhaley | you can add --add-cap's like you can to a runc I think | 15:55 |
| gwhaley | we generally don't recommend trying a --priviledge container though - it might run, but some things will be different from if you did that with runc | 15:55 |
| gwhaley | that is one of the places where it is hard for kata to match runc. and tbh, --privileged sort of goes against the idea of kata as a security enhancement :-) | 15:56 |
| gwhaley | as I asked earlier - are you sure you don't want to be trying all this in vanilla VMs, and not under kata? You seem to be trying to do things inside a kata container that you probably would not do in a normal container | 15:57 |
| errordeveloper | gwhaley: ah, so if I want priveleged pod, kata will read that from kubernetes, and it's going to be priveleged, but inside the kata VM? | 15:57 |
| errordeveloper | it's just something I didn't quite imagine, but totally make sense | 15:58 |
| gwhaley | wellll - generally speaking, yes, if you ask kata to make a priv pod, it will be priv inside the VM, and not to the host | 15:58 |
| gwhaley | but, also be aware that you can configure k8s so that priv pods run in runc, and non-priv run in kata - which might be more what one would expect/want | 15:58 |
| errordeveloper | for some reason it feels like the kata VM is still a bit of black box, I keep questioning and wondering about what goes on | 15:59 |
| *** devimc has quit IRC | 15:59 | |
| gwhaley | you should not need to know .... if you just want to run a container. but, well, it's a slimmed down VM that just has what is needed for kata to launch a libcontainer based container etc. | 15:59 |
| gwhaley | there are no *secrets* though - all the code is there, and we are happy to explain what and how it works | 15:59 |
| errordeveloper | I see, I just want to eliminate any doubts that my kata VM can run systemd+containerd+kubelete just fine, that's something I'm not sure of right now | 16:00 |
| errordeveloper | but I have containerd working now! =) | 16:00 |
| gwhaley | I'm just concerned you are trying to do quite non-container things in the belief that 'this is a VM' rather than 'this is a container' | 16:00 |
| gwhaley | but... (maybe I forgot if you stated this before...) - sure, if you have a goal that says 'can I run a systemd/kubelet/containerd inside a Kata container', then that is something that could be discussed. A github Issue might be the place to gather all the right folks thoughts | 16:02 |
| errordeveloper | gwhaley: sure, I rather know what I'm after here | 16:09 |
| gwhaley | ack, I see you have a lot of background in k8s, cncf, cilium etc..... | 16:11 |
| *** sgarzare has quit IRC | 16:23 | |
| *** jodh has quit IRC | 16:56 | |
| *** gwhaley has quit IRC | 17:02 | |
| *** Jean-Mick has joined #kata-dev | 17:26 | |
| *** Jean-Mick has quit IRC | 17:27 | |
| *** Jean-Mick has joined #kata-dev | 17:49 | |
| kata-irc-bot1 | <fidencio> @archana.m.shinde, seems that all the tests I've been doing with kata and OpenShift I've been blindly *not* using shimv2. In order to use shimv2 with OpenShift (well, with CRI-O in the end) is that this is what CRI-O needs: ```[crio.runtime.runtimes.kata-qemu]``` | 17:50 |
| Jean-Mick | Hello, from https://github.com/amshinde/kata-runtime/commit/4d470e513b31fb02bcfc4666f7976022a87e715b, it seems kata supports a way to override the default 64MB for /dev/shm, can someone confirm ? | 17:51 |
| kata-irc-bot1 | <fidencio> ```runtime_path = "/path/to/containerd-shim-kata-v2" runtime_type = "vm"``` | 17:51 |
| Jean-Mick | because I tried ¨- name: shm emptyDir: sizeLimit: "8Gi" medium: "Memory" but it does not work | 17:52 |
| kata-irc-bot1 | <archana.m.shinde> @fidencio Thats right, you need that configuration for shimv2 | 17:53 |
| kata-irc-bot1 | <fidencio> is there something else that has to be configured then on the kata configuration side? | 17:54 |
| kata-irc-bot1 | <archana.m.shinde> @fidencio Nope, thats the only configuration you need | 17:55 |
| kata-irc-bot1 | <archana.m.shinde> @Jean-Mick: what are you seeing? | 17:55 |
| kata-irc-bot1 | <fidencio> @archana.m.shinde Cool. And then I guess the whole `[shim.kata]` could be dropped from the configuration, right? | 17:55 |
| kata-irc-bot1 | <fidencio> BTW, sorry if the question is too silly, but what needs to be done on podman side so it'd be using shimv2 as well? | 17:56 |
| kata-irc-bot1 | <archana.m.shinde> @fidencio shimv2 requires 2 parts | 17:58 |
| kata-irc-bot1 | <archana.m.shinde> the server api and the client simplistically | 17:58 |
| kata-irc-bot1 | <archana.m.shinde> crio includes the server api | 17:58 |
| kata-irc-bot1 | <archana.m.shinde> podman would need the same | 17:59 |
| kata-irc-bot1 | <fidencio> Aha! Okay, okay. II remember Peter Hunt saying it wouldn't be too hard, but my knowledge around this area is, if not null, negative :slightly_smiling_face: | 18:17 |
| kata-irc-bot1 | <fidencio> @archana.m.shinde, thanks for the quick replies, really appreciated! | 18:17 |
| Jean-Mick | archana.m.shinde I still see 64m | 18:31 |
| kata-irc-bot1 | <archana.m.shinde> let me see whats going on | 18:43 |
| kata-irc-bot1 | <archana.m.shinde> We had tested the shmsize on docker context | 18:56 |
| kata-irc-bot1 | <archana.m.shinde> I havent tried yet, but I suspect this could be the reason :https://github.com/amshinde/kata-runtime/commit/4d470e513b31fb02bcfc4666f7976022a87e715b#diff-8af421444cdb401abb2498cf4c7a011eR631 | 18:57 |
| kata-irc-bot1 | <archana.m.shinde> removing the explicit check m.Source != "/dev/shm" | 18:57 |
| Jean-Mick | i will test that thanks a lot | 19:06 |
| Jean-Mick | newbie question sorry - after this patch, is it effective or a service should be restarted? | 19:07 |
| *** davidgiluk has quit IRC | 19:21 | |
| *** errordeveloper has quit IRC | 19:52 | |
| kata-irc-bot1 | <fidencio> What's the easiest / cheaper way to retrigger the travis-ci? | 20:17 |
| kata-irc-bot1 | <fidencio> seems that it either was not executed or got stuck in the SELinux PR | 20:18 |
| kata-irc-bot1 | <fidencio> @gabriela.cervantes.te ^ | 20:20 |
| kata-irc-bot1 | <salvador.fuentes> checking... | 20:23 |
| kata-irc-bot1 | <salvador.fuentes> github kind of slow and I got the unicorn...I have restarted on travis side, lets see if that helps | 20:24 |
| *** errordeveloper has joined #kata-dev | 20:26 | |
| kata-irc-bot1 | <fidencio> Thanks! I may need to restart the Ubuntu jobs later on, but some of the failures we're facing are differents from Yesterday's ones. At this point I guess it's safe to assume it's not related to the PR itself and have it merged (once the required CIs pass) | 20:26 |
| *** errordeveloper has quit IRC | 20:31 | |
| *** devimc has joined #kata-dev | 20:55 | |
| *** igordc has joined #kata-dev | 21:05 | |
| *** sameo has quit IRC | 21:14 | |
| *** crobinso has quit IRC | 21:15 | |
| *** errordeveloper has joined #kata-dev | 21:17 | |
| *** errordeveloper has quit IRC | 21:22 | |
| kata-irc-bot1 | <fidencio> @gabriela.cervantes.te, JFYI, could reproduce the issue using Fedora RPMS + vritiofsd | 21:25 |
| kata-irc-bot1 | <gabriela.cervantes.te> ohh ok | 21:26 |
| kata-irc-bot1 | <gabriela.cervantes.te> yes it is happening | 21:26 |
| kata-irc-bot1 | <fidencio> unfortunately I don't understand enough of the runtime code to dive into that quickly, but for whoever will work on that, I would be more than happy ensuring the patch works on Fedora *and* reviewing the code | 21:28 |
| *** errordeveloper has joined #kata-dev | 21:30 | |
| *** ailan has quit IRC | 21:39 | |
| *** devimc has quit IRC | 22:16 | |
| *** errordeveloper has quit IRC | 22:33 | |
| *** openstack has joined #kata-dev | 22:57 | |
| *** ChanServ sets mode: +o openstack | 22:57 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!