| *** kgz has quit IRC | 00:11 | |
| *** kgz has joined #kata-dev | 00:23 | |
| *** gmmaharaj has joined #kata-dev | 00:32 | |
| *** gmmaha has quit IRC | 00:32 | |
| *** gmmaharaj is now known as gmmaha | 00:32 | |
| kata-irc-bot | <xwlpt> Hi, it seems it is not supported to get the agent log when kata works at shimv2+vsock mode. For other work mode(build in or not ) which need proxy, there is a go routine to watch the console to get the log, but for no proxy vsock mode, there is no such go routine. | 01:56 |
|---|---|---|
| kata-irc-bot | <xwlpt> Besides, do we have the docs about the details of the network working mode for kata, macvtap , bridged , tcfilter etc. | 01:57 |
| kata-irc-bot | <xwlpt> https://github.com/kata-containers/documentation/blob/master/design/architecture.md#networking The working mode should be bridged but not macvtap. | 01:59 |
| kata-irc-bot | <xwlpt> cc @fupan @eric.ernst | 02:08 |
| kata-irc-bot | <eric.ernst> Hey @xwlpt | 02:18 |
| kata-irc-bot | <eric.ernst> That's correct. You'd need to start using opentracing instead. | 02:18 |
| kata-irc-bot | <eric.ernst> IF this isn't documented well, we should open up an issue on github.com/kata-containers/documentation... | 02:19 |
| kata-irc-bot | <xwlpt> @eric.ernst Thanks. I supposed that it support to get the opentracing and console log either. Will have a try of opentracing. Thanks | 02:20 |
| kata-irc-bot | <eric.ernst> np. It isn't as straight forward is it should be today -- @james.o.hunt is putting alot of work into here to help improve it. | 02:21 |
| kata-irc-bot | <eric.ernst> Please share your experience with enabling this and don't hestitate to reach out if you run into issues. | 02:21 |
| kata-irc-bot | <xwlpt> Sure, thanks @eric.ernst | 02:21 |
| *** igordc has quit IRC | 02:47 | |
| *** sameo has joined #kata-dev | 04:39 | |
| *** sgarzare has joined #kata-dev | 06:17 | |
| *** sgarzare has quit IRC | 06:21 | |
| *** sgarzare has joined #kata-dev | 06:47 | |
| *** sameo has quit IRC | 06:54 | |
| *** tmhoang has joined #kata-dev | 06:55 | |
| *** kgz has quit IRC | 07:20 | |
| *** kgz has joined #kata-dev | 07:28 | |
| *** sameo has joined #kata-dev | 07:31 | |
| *** jodh has joined #kata-dev | 07:36 | |
| *** davidgiluk has joined #kata-dev | 08:03 | |
| *** gwhaley has joined #kata-dev | 08:03 | |
| brtknr | Anyone know why I'm hitting this: unable to recognize "kata-qemu-runtimeClass.yaml": no matches for kind "RuntimeClass" in version "node.k8s.io/v1beta1" | 08:10 |
| brtknr | This worked on minikube | 08:10 |
| brtknr | Im using k8s 1.14 | 08:10 |
| brtknr | With cri-o | 08:11 |
| gwhaley | brtknr: my best guess is different k8s versions - the name/status of RuntimeClass transitioned form alpha to beta between 1.13.4 and 1.14 . minikube updated to 1.14 about 1.5weeks ago or so... | 08:24 |
| gwhaley | let me find a PR that shows the differences to our yaml: | 08:24 |
| brtknr | gwhaley, i know what the problem is... i used kubespray to upgrade my k8s version and it seems to not have done it properly | 08:25 |
| brtknr | i was at your workshop at openinfra days :) | 08:25 |
| gwhaley | brtknr: heh, I did wonder if you were there ;-) OK, will leave you with your spraying then.... | 08:25 |
| brtknr | gwhaley: its nice that runtime class crd is no longer required in 1.14 | 08:30 |
| brtknr | gwhaley: am I right in thinking thta? | 08:30 |
| brtknr | s/thta/that | 08:30 |
| gwhaley | brtknr: I believe that is correct - somebody on the course checked and it was already there in 1.14 - but, I have not checked myself... :-) | 08:41 |
| brtknr | gwhaley: Must I use cri-o runtime or can i also use docker? | 09:12 |
| *** EricRen has joined #kata-dev | 09:13 | |
| brtknr | kubeadm is refusing to start for k8s version 1.14 :( | 09:16 |
| gwhaley | brtknr: I think you need to use either cri-o or containerd to enable kata under k8s. You need something that knows about RuntimeClass - just docker does not do that iiuc | 09:24 |
| *** lpetrut has joined #kata-dev | 10:27 | |
| brtknr | does qemu need to be present in workers for runtime class to work? | 10:44 |
| *** changcheng has joined #kata-dev | 10:44 | |
| brtknr | or does kata-deploy automatically install that? | 10:44 |
| brtknr | is it better to use qemu-system-x86_64 or qemu-lite-system-x86_64? | 10:45 |
| gwhaley | brtknr: kata-deploy installs it, in /opt/kata/bin/* and other bits under /opt/kata :-) | 10:47 |
| gwhaley | you can use either - qemu-lite should be a little smaller/faster. We also work with distro provided qemu's, as some distros will not want to install a custom qemu-lite as well as their distro provided one | 10:48 |
| gwhaley | the kata installed by kata-deply should be pre-configured to point to the qemu it installed. | 10:48 |
| gwhaley | if you run '/opt/kata/bin/kata-qemu kata-env', it will show you how it is configured for instance | 10:49 |
| brtknr | gwhaley: hmm its appears to be spinning up regular runc containers | 10:52 |
| brtknr | gwhaley: what happens if kvm_intel is not enabled? | 10:54 |
| brtknr | http://paste.openstack.org/show/749170/ | 10:56 |
| brtknr | [centos@kata-worker-1 ~]$ cat /sys/module/kvm_intel/parameters/nested | 10:56 |
| brtknr | Y | 10:56 |
| brtknr | I modprobed vhost and vhost_net and now I get that my system is capable of running kata containers | 11:06 |
| gwhaley | brtknr: yay! - yes, those vhost items are needed. I suspect you would have found errors in the logs when you were trying to run kata containers without them. | 11:07 |
| *** gwhaley has quit IRC | 11:07 | |
| *** devimc has joined #kata-dev | 11:51 | |
| brtknr | Do I also need to install qemu-kvm on the worker nodes separately? | 11:58 |
| *** gwhaley has joined #kata-dev | 12:06 | |
| gwhaley | brtknr: I think the qemu-kvm package just provides the distro shipped qemu-system-x86_64 and all its bits, does it not? In which case, no, the kata-deploy installs all the qemu bits you need for kata down in /opt/kata . For instance, I'm pretty sure I didn't have that package installed on the minikube I was using | 12:09 |
| brtknr | gwhaley: I must be missing something... because I can still see the container under crictl | 12:10 |
| brtknr | And I dont see anything when i grep for qemu under ps -ef | 12:11 |
| brtknr | Apart from the log services | 12:11 |
| gwhaley | brtknr: I dunno - probably best approach is to look at the various logs to try and see what went wrong. there is a journalctl example on https://github.com/kata-containers/tests/blob/master/.ci/teardown.sh#L77 for instance. If that all fails, maybe open a github Issue and paste the requested info there. You've probably hit the limits of my k8s knowledge, and we'll need others for some input. | 12:16 |
| brtknr | Hard to describe the issue when I dont know waht the issue is lol :P | 12:19 |
| gwhaley | well, the issue you say is `failed to launch a kata runtime container under k8s`, no? :-) | 12:20 |
| gwhaley | If you have a look in that runtime journalctl log, hopefully there is a line that says 'failed to launch container' or similar, and a reason... and then maybe we can go from there. | 12:20 |
| *** changcheng has quit IRC | 13:40 | |
| *** EricRen has quit IRC | 13:48 | |
| *** changcheng has joined #kata-dev | 13:51 | |
| *** lpetrut has quit IRC | 14:19 | |
| *** EricRen has joined #kata-dev | 14:29 | |
| *** dklyle has joined #kata-dev | 14:51 | |
| *** dhellmann has joined #kata-dev | 15:10 | |
| *** tmhoang has quit IRC | 15:14 | |
| stefanha | "rpc error: code = Unknown desc = selinux label is specified in config, but selinux is disabled or not supported" <-- from kata-agent | 15:19 |
| stefanha | Any idea how to solve this issue? My docker command-line is docker run --runtime=kata-runtime busybox sh. | 15:19 |
| stefanha | I don't have any explicit SELinux labels defined. | 15:19 |
| kata-irc-bot | <graham.whaley> feels like you maybe dont' have the selinux libs in the rootfs of the VM? | 15:20 |
| kata-irc-bot | <graham.whaley> but, have built with selinux enabled in osbuilder or agent build - my guess | 15:20 |
| stefanha | graham.whaley: It's a ClearLinux initramfs. | 15:20 |
| kata-irc-bot | <graham.whaley> cc @salvador.fuentes @jose.carlos.venegas.m (@niteshkonkar007 who wrote the code iirc) | 15:20 |
| stefanha | graham.whaley: Thanks, I'll check if I can rebuild without SELinux support. | 15:20 |
| kata-irc-bot | <graham.whaley> I have a feeling we build with it off by default, as it is costly (to size and speed)... | 15:21 |
| stefanha | graham.whaley: I'm fine with that. But something is sneaking in ProcessLabel in the config. I'm not sure how to disable that. | 15:22 |
| kata-irc-bot | <eric.ernst> It is off by default | 15:22 |
| stefanha | Maybe it's my Fedora Docker default config that is SELinux-happy. | 15:22 |
| stefanha | I haven't found a way to disable that in docker-run(1) though :( | 15:22 |
| kata-irc-bot | <eric.ernst> Folks can always add. I know we had talks about this in the past; it’s just for extra paranoid security between containers in the same pod | 15:22 |
| kata-irc-bot | <eric.ernst> I don’t know where dan Walsh is, so I am afraid of saying set enforce 0 :) | 15:23 |
| kata-irc-bot | <graham.whaley> stefanha - ouch, so, indeed, it seems if you have selinux on in your global docker config, it is on for all containers/runtimes... afaict. so, iswym | 15:26 |
| kata-irc-bot | <graham.whaley> I guess you can test with Erics enforce0 hack... | 15:26 |
| kata-irc-bot | <graham.whaley> and rebuild the agent and rootfs/image.kernel with selinux enabled if you need to enable it.. | 15:26 |
| kata-irc-bot | <graham.whaley> <thanks docker...> | 15:26 |
| stefanha | I tried setenforce 0 but it doesn't help. I guess that's because this is about SELinux inside the container not on the host. | 15:29 |
| stefanha | Maybe I can figure out the Docker magic to clear that option even if it has been set by default on Fedora. | 15:30 |
| * stefanha puts on his "setenforce 0" T-shirt | 15:30 | |
| kata-irc-bot | <graham.whaley> you might be able to set it in your local user docker config, or in /etc/docker/*... | 15:32 |
| kata-irc-bot | <graham.whaley> stefanha: problaby look for who is adding `--selinux-enabled` to the dockerd command line is my guess.... either a config file or the systemd command line. | 15:34 |
| stefanha | Thanks, I see it's enabled by default in /etc/sysconfig/docker. | 15:35 |
| * stefanha tries to disable it | 15:35 | |
| *** eernst has joined #kata-dev | 15:48 | |
| *** sameo has quit IRC | 15:54 | |
| *** EricRen has quit IRC | 16:21 | |
| brtknr | is there a repo for 1.14 or does it have to be installed from source? | 16:22 |
| gwhaley | brtknr: k8s, or cri-o, or kata or ? :-) /cc fuentess | 16:24 |
| brtknr | cri-o oops | 16:24 |
| brtknr | gwhaley: ^ | 16:25 |
| kata-irc-bot | <salvador.fuentes> I think you will need to build from sources, /cc @sebastien.boeuf ^ | 16:25 |
| brtknr | thank you | 16:27 |
| kata-irc-bot | <raravena80> I think there are no cri-o pkgs for 1.14 yet. https://launchpad.net/~projectatomic/+archive/ubuntu/ppa | 16:27 |
| kata-irc-bot | <raravena80> Regarding selinux _set enforce 0_ :+1: :+1: | 16:28 |
| gwhaley | fuentess, jcvenega - this may not be the reason, but I just re-created the '1 != 0, timeout' make functional failure inside a ccloudvm ubunutu 18.... the reason for this fail is... | 16:31 |
| gwhaley | that the ccloudvm does not have enough RAM... | 16:31 |
| gwhaley | so, maybe that is not what kills us in the CI - but, thought I should note... | 16:31 |
| gwhaley | I see the stderr output (thansk jcvenega), and it is qemu complaining about -m option and sizes | 16:32 |
| gwhaley | let me bump my ccloudvm RAM and try again .. | 16:32 |
| kata-irc-bot | <salvador.fuentes> we have ~16GB of memory in the CI VMs, so yeah, should be a different reason | 16:33 |
| gwhaley | yeah, bumped the RAM and the tests now pass. Let's see if we get some stdout on the next fail in the CI.. | 16:36 |
| kata-irc-bot | <sebastien.boeuf> yeah build crio from the 1.14 branch | 16:36 |
| *** sameo has joined #kata-dev | 16:47 | |
| brtknr | gwhaley: in the workshop we did, the version of cri-o was 1.13 | 16:50 |
| brtknr | and it still worked | 16:50 |
| brtknr | @sebastien.boeuf ^^ | 16:50 |
| kata-irc-bot | <sebastien.boeuf> brtknr: sorry what's the question? | 16:51 |
| gwhaley | brtknr: I ... dunno. It was whatever was in k8s v1.14 (in minikube). I also used k8s v1.13.4, and that worked.... | 16:51 |
| gwhaley | so, I think the runtimeclass came in at v1.12 iirc | 16:51 |
| gwhaley | sboeuf: do k8s release numbers align with cri-o release numbers? | 16:52 |
| brtknr | I found a centos repo where its available to download via yum: https://cbs.centos.org/repos/paas7-crio-113-candidate/x86_64/os/ | 16:52 |
| gwhaley | in 1.13, it will still be alpha, so you will have to apply the gate enabling | 16:52 |
| gwhaley | in 1.14 it became beta, so is on by default | 16:52 |
| brtknr | gwhaley: you're talking about k8s though right? | 16:53 |
| brtknr | I'm talking about cri-o | 16:53 |
| brtknr | aaaah | 16:53 |
| gwhaley | I am (I don't know if they align or not on version numbers...) | 16:53 |
| *** sgarzare has quit IRC | 17:03 | |
| *** gwhaley has quit IRC | 17:03 | |
| *** jodh has quit IRC | 17:03 | |
| *** igordc has joined #kata-dev | 17:06 | |
| brtknr | gwhaley: seen this? | 17:37 |
| brtknr | Warning FailedCreatePodSandBox 2s kubelet, minikube Failed create pod sandbox: rpc error: code = Unknown desc = container create failed: Failed to check if grpc server is working: rpc error: code = Unavailable desc = transport is closing | 17:37 |
| brtknr | I'm attempting to recreate the minikube example from the workshop... | 17:39 |
| *** davidgiluk has quit IRC | 19:05 | |
| *** sameo has quit IRC | 20:07 | |
| *** fuentess has quit IRC | 21:04 | |
| *** devimc has quit IRC | 21:12 | |
| *** igordc has quit IRC | 22:09 | |
| *** eernst has quit IRC | 23:04 | |
| *** eernst has joined #kata-dev | 23:08 | |
| *** eernst has quit IRC | 23:13 | |
| *** changcheng has quit IRC | 23:21 | |
| *** noahm has quit IRC | 23:21 | |
| *** changcheng has joined #kata-dev | 23:27 | |
| *** noahm has joined #kata-dev | 23:27 | |
| kata-irc-bot | <salvador.fuentes> hi all, upgrading jenkins, will be offline some moments | 23:38 |
| *** eernst has joined #kata-dev | 23:40 | |
| *** igordc has joined #kata-dev | 23:44 | |
| *** eernst has quit IRC | 23:45 | |
| *** eernst has joined #kata-dev | 23:45 | |
| *** eernst has quit IRC | 23:46 | |
| *** eernst_ has joined #kata-dev | 23:46 | |
| *** eernst_ has quit IRC | 23:51 | |
| kata-irc-bot | <salvador.fuentes> jenkins is back | 23:55 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!