| kata-irc-bot | <verytired1> @eric.ernst i think we might have grabbed the katacontainers namespace on docker hub. let me see if we did | 00:50 |
|---|---|---|
| kata-irc-bot | <krsna1729> there does exist a page https://hub.docker.com/u/katacontainers | 01:30 |
| *** zerocoolback has joined #kata-dev | 03:01 | |
| *** sjas has joined #kata-dev | 04:18 | |
| *** sjas_ has quit IRC | 04:21 | |
| kata-irc-bot | <caihaomin> Can find CI retrigger button after update... where did it go buddies~~~ @sebastien.boeuf @james.o.hunt | 05:16 |
| *** Ankita_ has joined #kata-dev | 05:20 | |
| *** Ankita_ has quit IRC | 06:16 | |
| *** jodh has joined #kata-dev | 07:16 | |
| *** jodh has quit IRC | 07:16 | |
| *** jodh has joined #kata-dev | 07:16 | |
| kata-irc-bot | <james.o.hunt> @caihaomin - I see a "rebuild" button when I login (click top right button) on http://jenkins.katacontainers.io/. If you don't have that option, please ping @salvador.fuentes / @mnaser. | 07:25 |
| *** zerocoolback has quit IRC | 07:25 | |
| *** sameo has joined #kata-dev | 07:25 | |
| *** zerocoolback has joined #kata-dev | 07:30 | |
| *** zerocoolback has quit IRC | 07:35 | |
| kata-irc-bot | <eric.ernst> Thanks @verytired1 | 07:50 |
| *** davidgiluk has joined #kata-dev | 08:02 | |
| kata-irc-bot | <caihaomin> @james.o.hunt thx and then ping @salvador.fuentes Plz help me solve this ~~~ :grinning: | 08:03 |
| *** gwhaley has joined #kata-dev | 08:04 | |
| *** zerocoolback has joined #kata-dev | 08:05 | |
| *** eernst has joined #kata-dev | 08:16 | |
| kata-irc-bot | <niteshkonkar007> Hey, do we pass rootfs using DAX to our kata-container? Also if I understand correctly, the rootfs approach is basically making a qemu-img disk, copy rootfs into it, install gpart etc binaries into it, pass it to the VM using DAX. Why do we not include kernel inside our rootfs image? Why do we pass it seperately? | 08:34 |
| *** zerocoolback has quit IRC | 08:37 | |
| *** zerocoolback has joined #kata-dev | 08:44 | |
| stefanha | niteshkonkar007: Maybe boot-from-NVDIMM doesn't work, I don't remember if the guest firmware supports it. | 08:55 |
| stefanha | niteshkonkar007: If you tell the Kata Runtime to use a disk image then it should use NVDIMM (DAX) for the rootfs. | 08:56 |
| stefanha | niteshkonkar007: Make sure your configuration.toml file has the path to the disk image. | 08:57 |
| kata-irc-bot | <niteshkonkar007> Yes, I know the approach of using rootfs instead of initrd. I was curious to know how the rootfs approach works in kata Vs the initrd approach(not enuf documentation on it) and why do we pass the kernel seperately and as part of the rootfs disk. | 08:59 |
| stefanha | niteshkonkar007: Like I said, the kernel is probably separate because guest firmware cannot boot directly from NVDIMM. | 09:03 |
| stefanha | niteshkonkar007: Regarding disk image vs initramfs. On a host running multiple Kata VMs the disk image will be shared in the host page cache. | 09:03 |
| *** zerocoolback has quit IRC | 09:04 | |
| stefanha | If you use an initramfs then the memory is duplicated until Kernel Samepage Merging decides to deduplicate it. | 09:04 |
| stefanha | niteshkonkar007: So there could be memory savings if you using a disk image. | 09:04 |
| *** zerocoolback has joined #kata-dev | 09:11 | |
| *** zerocoolback has quit IRC | 10:06 | |
| *** eernst has quit IRC | 10:08 | |
| kata-irc-bot | <niteshkonkar007> @stefanha: Thanks a lot for the info. | 10:38 |
| *** zerocoolback has joined #kata-dev | 10:54 | |
| *** gwhaley has quit IRC | 11:02 | |
| *** devimc has joined #kata-dev | 11:54 | |
| *** gwhaley has joined #kata-dev | 12:18 | |
| *** fuentess has joined #kata-dev | 12:23 | |
| kata-irc-bot | <salvador.fuentes> ping @caihaomin, hi, can you give me your github user so I can add the rebuild button for your user? | 12:26 |
| *** dklyle_ has joined #kata-dev | 12:57 | |
| *** manjeets_ has joined #kata-dev | 12:58 | |
| *** david-lyle has quit IRC | 12:59 | |
| *** manjeets has quit IRC | 13:00 | |
| *** LinuxMe has joined #kata-dev | 13:11 | |
| *** lamego has joined #kata-dev | 13:33 | |
| *** zerocoolback has quit IRC | 13:40 | |
| *** zerocoolback has joined #kata-dev | 13:40 | |
| gwhaley | hey folks - in case any of the github org owners were wondering, that jenkins app that just requested org access was myself and @salvador.fuentess sorting our more wrinkles in the new CI | 13:41 |
| *** zerocoolback has quit IRC | 13:42 | |
| *** zerocoolback has joined #kata-dev | 13:45 | |
| *** isaagar has quit IRC | 13:57 | |
| *** zerocoolback has quit IRC | 14:04 | |
| *** isaagar has joined #kata-dev | 14:06 | |
| *** annabelleB has joined #kata-dev | 14:11 | |
| *** annabelleB has quit IRC | 14:30 | |
| kata-irc-bot | <caihaomin> @salvador.fuentes jshachm thx a lot | 14:37 |
| kata-irc-bot | <salvador.fuentes> @caihaomin can you check if you already have permissions? | 14:38 |
| clarkb | if anyone has a moment this morning, it would be helpful to understand why the cpu allotment tests seem to check for N+1 CPUs instead of just N cpus and pass under jenkins but fail under zuul. | 14:40 |
| clarkb | http://logs.openstack.org/74/74/3250f409797a162789a5078496b6977caaf319e8/third-party-check/kata-runsh/873d8f8/job-output.txt.gz#_2018-06-15_23_22_03_773254 shows the test for 2 cpus failing because it is actually looking for 3 cpus (cpus are zero indexed so cpu2 is third cpu) | 14:40 |
| clarkb | http://jenkins.katacontainers.io/job/kata-containers-tests-centos-7-4-master/3/console shows this test checking for 3 cpus instead of 2 under jenkins and passing (which makes it extra confusing) | 14:40 |
| *** annabelleB has joined #kata-dev | 14:42 | |
| kata-irc-bot | <caihaomin> It works thanks @salvador.fuentes | 14:46 |
| kata-irc-bot | <salvador.fuentes> @julio.montes ^^ | 14:47 |
| *** dklyle_ has quit IRC | 14:49 | |
| devimc | clarkb, Hi | 14:51 |
| devimc | clarkb, https://github.com/kata-containers/documentation/blob/master/constraints/cpu.md | 14:51 |
| *** sameo has quit IRC | 14:51 | |
| devimc | https://github.com/kata-containers/documentation/blob/master/constraints/cpu.md#do-not-waste-resources | 14:52 |
| clarkb | devimc: (30000 + (20000 -1)) / 20000 = 2.49995 and the test is for 2 vcpus which seems to line up, but why check for 3 cpus in that case? | 14:56 |
| clarkb | depends on rounding maybe? | 14:56 |
| *** manjeets_ is now known as manjeets | 14:58 | |
| devimc | (30000 + (20000 -1)) / 20000 = 2.49995 = 2vCPUs | 14:59 |
| devimc | clarkb, runtime will hot add 2 vCPUs | 14:59 |
| devimc | but the VM already have 1 vCPUs | 14:59 |
| clarkb | oh its addin that absolute value not bringing the total available to that value? | 14:59 |
| devimc | yes | 15:00 |
| clarkb | ok that explains my confusion. Should probably update the test labels at the very least (it is checking for 3 cpus not 2) | 15:00 |
| clarkb | doesn't explain why it fails under zuul though | 15:01 |
| clarkb | which log files would be helpful to debug that? docker and kata runtime? | 15:01 |
| devimc | clarkb, runtime and agent | 15:02 |
| devimc | sudo journalctl -b -t kata-proxy | 15:02 |
| devimc | sudo journalctl -b -t kata-runtime | 15:02 |
| clarkb | thanks, I'll start by grabbing that info | 15:02 |
| clarkb | (just thinking out loud here, it is odd to me that if you set the quota on an instance you would get more than that quota, as a user I would probably find this unintuitive. I would expect quota to be absolute not additive) | 15:09 |
| devimc | clarkb, why? | 15:17 |
| clarkb | devimc: because quota is an explicit limit, if I set a quota that should be the limit | 15:17 |
| devimc | clarkb, yes, but you can also update that quota | 15:18 |
| clarkb | yup and that will be the new limit. The issue is that the limit I am setting is not the limit that is used | 15:18 |
| devimc | really? | 15:19 |
| devimc | that's weird | 15:19 |
| clarkb | devimc: yes instead it is the quota I set + some default configuration value of the service I may not even know about (if I am consuming the service and not administrator of it) | 15:19 |
| clarkb | re journalctl about did you mean kata-runtime and kata-agent instead of kata-runtime and kata-proxy? or is the proxy equivalent to the agent for logging? | 15:20 |
| devimc | kata-proxy contains agent logs | 15:21 |
| devimc | clarkb, but you have to add agent.log=debug as kernel parameter | 15:21 |
| devimc | in runtime configuration file | 15:21 |
| devimc | kernel_params = "agent.log=debug" | 15:21 |
| clarkb | devimc: I'm just running .ci/setup.sh && .ci/run.sh fwiw | 15:22 |
| devimc | ohh | 15:22 |
| devimc | clarkb, please edit runtime configuration file | 15:23 |
| devimc | and enable debugging | 15:23 |
| devimc | \/usr/share/defaults/kata-containers/configuration.toml | 15:23 |
| clarkb | will do thanks | 15:23 |
| devimc | clarkb, is http://logs.openstack.org/74/74/3250f409797a162789a5078496b6977caaf319e8/third-party-check/kata-runsh/873d8f8/job-output.txt.gz#_2018-06-15_23_29_31_744467 your log file? | 15:24 |
| clarkb | devimc: yes | 15:24 |
| devimc | uhmmm I see | 15:25 |
| *** dklyle has joined #kata-dev | 15:29 | |
| *** fiddletwix has joined #kata-dev | 15:37 | |
| clarkb | devimc: do you see something odd other than the tests failing? | 15:47 |
| clarkb | looks like it claims to enable all debug options in the config file already, I'm just going to make sure I collect the data | 15:47 |
| devimc | clarkb, yep | 15:48 |
| devimc | command failed error 'exit status 1' | 15:49 |
| devimc | seems like vCPUs are not connected | 15:49 |
| devimc | would you mind modifying a test and re-run them? | 15:49 |
| clarkb | devimc: or not enough of them were connected. Another update may be to run nproc before the sysfs check (so that we can see it) | 15:50 |
| clarkb | devimc: ya once I've got log collection going I think I will push an update to the test side to see if we can get more info there as well like ^ | 15:50 |
| kata-irc-bot | <mnaser> fyi: clarkb is using the same exact vms we're using in jenkins.katacontainers.io | 15:51 |
| kata-irc-bot | <mnaser> so something to keep in mind | 15:51 |
| devimc | clarkb, ok, try modifying https://github.com/kata-containers/tests/blob/master/integration/docker/cpu_test.go#L246 | 15:51 |
| clarkb | mnaser are you using our images? this is running on infra images which may be different | 15:51 |
| devimc | increase that number | 15:51 |
| clarkb | flavor is the same though | 15:51 |
| devimc | 10 or 15 | 15:51 |
| kata-irc-bot | <mnaser> clarkb: images are different, using 100% 'upstream' images | 15:51 |
| kata-irc-bot | <mnaser> also, is this still on bionic or on xenial? seems xenial is the stablest of the bunch | 15:52 |
| clarkb | xenial, switched when bionic wasn't working due to lack of docker-ce packages | 15:52 |
| devimc | clarkb, FOCUS="Update number of CPUs" RUNTIME=kata-runtime make -e integration | 15:52 |
| clarkb | I've asked it for a new run where it should collect logs, will see what that tells us then do test updates if necessary | 15:56 |
| clarkb | thank you for the help | 15:57 |
| kata-irc-bot | <salvador.fuentes> @mnaser, can you provide me a VM with 17.10 to check if the script I created can hang the machine? | 15:58 |
| kata-irc-bot | <mnaser> oh yes | 15:59 |
| kata-irc-bot | <mnaser> sure | 15:59 |
| *** annabelleB has quit IRC | 16:00 | |
| kata-irc-bot | <mnaser> @salvador.fuentes rebuild the same machine on same ip -- 162.253.55.170 | 16:00 |
| kata-irc-bot | <salvador.fuentes> thanks @mnaser | 16:01 |
| devimc | clarkb, how many CPUs do you have? | 16:01 |
| devimc | you need at least 4 | 16:02 |
| clarkb | devimc: should be 8, it is the same flavor in vexxhost that jenkins is using | 16:02 |
| clarkb | devimc: but I can double check that as well | 16:02 |
| *** annabelleB has joined #kata-dev | 16:02 | |
| devimc | clarkb, how much RAM? | 16:03 |
| clarkb | devimc: 16GB iirc, mnaser is that right? | 16:04 |
| kata-irc-bot | <mnaser> devimc, clarkb: correct, same *exact* flavor | 16:14 |
| devimc | mnaser, ok | 16:18 |
| devimc | clarkb, probably you are doing something different | 16:19 |
| clarkb | devimc: right now my hunch is something different in the images. Possibly a missing kernel module or different kernel version? On the job execution side I'm basically just installing build-essential, golang, and docker then running .ci/setup.sh && .ci/run.sh | 16:21 |
| devimc | clarkb, what version of docker? | 16:23 |
| clarkb | devimc: oh sorry I don't install docker anymore, I was on bionic, but switched to xenial so that setup.sh can do it | 16:23 |
| kata-irc-bot | <mnaser> clarkb: in ci, we don't rely on setup.sh to install docker | 16:24 |
| kata-irc-bot | <mnaser> not sure if that has to do with anything | 16:24 |
| *** gwhaley has quit IRC | 16:49 | |
| kata-irc-bot | <anne> Would folks ping me or reply here if they intended to come to the Kata in-person meeting in September in Denver? (trying to get an approximate head count) | 17:04 |
| *** jodh has quit IRC | 17:11 | |
| *** annabelleB has quit IRC | 17:17 | |
| *** annabelleB has joined #kata-dev | 17:22 | |
| *** sameo has joined #kata-dev | 17:27 | |
| *** dklyle has quit IRC | 17:36 | |
| *** dklyle has joined #kata-dev | 17:37 | |
| clarkb | devimc: Jun 18 17:23:18.467727 ubuntu-xenial-vexxhost-vexxhost-ca-ymq-1-0000202446 kata-runtime[22561]: time="2018-06-18T17:23:18.467708584Z" level=debug msg="the actual number of vCPUs will not be modified" arch=amd64 container-id=67cdc1f89636ef60a6e99aabade0105f213c12603ec176aef5ebde4c3bee1edd name=kata-runtime new-vcpus=0 old-vcpus=0 pid=22561 | 17:48 |
| clarkb | sandbox-id=67cdc1f89636ef60a6e99aabade0105f213c12603ec176aef5ebde4c3bee1edd source=virtcontainers subsystem=container | 17:48 |
| clarkb | devimc: seems to correspond to http://logs.openstack.org/74/74/8593bec79c873812b772d1de1d77ed63ebcb7c83/third-party-check/kata-runsh/44260b0/job-output.txt.gz#_2018-06-18_17_23_18_433067 | 17:48 |
| clarkb | devimc: other logs can be found at http://logs.openstack.org/74/74/8593bec79c873812b772d1de1d77ed63ebcb7c83/third-party-check/kata-runsh/44260b0/ and I think I held the node and shoul dhave access to it if we want to do more debugging | 17:49 |
| clarkb | https://github.com/kata-containers/runtime/blob/b3b0612fbeab7d89505d98beea6af609c0addbad/virtcontainers/container.go#L1160-L1165 produces that message | 17:50 |
| clarkb | the value of 0 is special right? does this imply a configuration difference? | 17:50 |
| clarkb | mnaser ^ you may know | 17:50 |
| devimc | clarkb, I don't see agent logs | 17:52 |
| devimc | did you enable it? | 17:52 |
| clarkb | devimc: no, because setup.sh said it was enabling all debug flags in the configuration anyway (I guess it could be failing to do that for the proxy?) | 17:52 |
| clarkb | devimc: but it seems the runtime is shortcircuiting due to the value being 0 | 17:52 |
| clarkb | http://logs.openstack.org/74/74/8593bec79c873812b772d1de1d77ed63ebcb7c83/third-party-check/kata-runsh/44260b0/job-output.txt.gz#_2018-06-18_17_15_58_033361 is why I didn't update the config | 17:53 |
| clarkb | I'm checking directly on the test node now | 17:54 |
| clarkb | kernel_params = " agent.log=debug" is set | 17:55 |
| clarkb | default_vcpus = 1 as well | 17:55 |
| devimc | clarkb, thta's weird | 17:56 |
| devimc | level=debug msg="the actual number of vCPUs will not be modified" arch=amd64 container-id=67cdc1f89636ef60a6e99aabade0105f213c12603ec176aef5ebde4c3bee1edd name=kata-runtime new-vcpus=0 old-vcpus=0 | 17:56 |
| devimc | new-vcpus=0 old-vcpus=0 | 17:56 |
| devimc | so, seems like you are trying to add 0 vcpus | 17:57 |
| clarkb | devimc: ya and the old value is 0 as well (I would expect it to be 1) | 17:57 |
| devimc | no, that's ok | 17:57 |
| devimc | 0 is because the container has not vcpus assigned | 17:58 |
| devimc | seems like docker is not giving the right number of vcpus to the runtime | 17:59 |
| clarkb | ah | 17:59 |
| * clarkb looks at docker log | 17:59 | |
| devimc | probably you are installing an old version of docker | 17:59 |
| devimc | o_O ? | 17:59 |
| clarkb | maybe? it is just using setup.sh for that. Let me find the log of that | 18:00 |
| clarkb | http://logs.openstack.org/74/74/8593bec79c873812b772d1de1d77ed63ebcb7c83/third-party-check/kata-runsh/44260b0/job-output.txt.gz#_2018-06-18_17_11_06_279339 17.12.1 ? | 18:00 |
| *** dklyle has quit IRC | 18:00 | |
| clarkb | Docker version 17.12.1-ce, build 7390fc6 says `docker --version` | 18:01 |
| devimc | clarkb, hehe yep | 18:01 |
| devimc | that's the error | 18:01 |
| devimc | that version of docker is broken | 18:01 |
| clarkb | ok, that is what setup.sh provides... | 18:02 |
| devimc | I don't know | 18:02 |
| devimc | please update it manually | 18:02 |
| devimc | > 18 | 18:02 |
| clarkb | ok let me see how to do that and then update .ci/setup.sh to do it | 18:03 |
| devimc | clarkb, http://jenkins.katacontainers.io/job/kata-containers-runtime-ubuntu-16-04-PR/37/consoleFull | 18:03 |
| devimc | Add runtime as a new/default Docker runtime. Docker version "Docker version 18.03.1-ce, build 9ee9f40" could change according to updates. | 18:03 |
| devimc | I don't know what's going on | 18:04 |
| devimc | it's xenial but with docker >=18 | 18:05 |
| clarkb | devimc: ya mnaser mentioned earlier that the jenkins jobs install docker out of band. I think the proper fix here is to update .ci/setup.sh to do the thing that works then stop jenkins from overriding it at all | 18:05 |
| clarkb | devimc: https://github.com/kata-containers/runtime/blob/master/versions.yaml#L130 is the problem | 18:13 |
| devimc | clarkb, you're right | 18:14 |
| *** dklyle has joined #kata-dev | 18:26 | |
| clarkb | devimc: https://review.openstack.org/576178 is my current state. I will rerun that to confirm it fixes things, then we can figure out how to fix .ci/setup.sh so that jenkins (and anything else) can run it and test it | 18:41 |
| devimc | clarkb, cool :) | 18:51 |
| *** davidgiluk has quit IRC | 19:19 | |
| *** annabelleB has quit IRC | 19:27 | |
| *** annabelleB has joined #kata-dev | 19:32 | |
| *** LinuxMe has quit IRC | 19:48 | |
| *** LinuxMe has joined #kata-dev | 19:49 | |
| *** devimc has quit IRC | 19:50 | |
| *** LinuxMe has quit IRC | 19:54 | |
| *** sameo has quit IRC | 20:01 | |
| *** LinuxMe_ has joined #kata-dev | 20:02 | |
| *** annabelleB has quit IRC | 20:42 | |
| *** annabelleB has joined #kata-dev | 20:43 | |
| *** annabelleB has quit IRC | 20:54 | |
| *** marst has joined #kata-dev | 21:17 | |
| *** fuentess has quit IRC | 21:26 | |
| *** LinuxMe_ has quit IRC | 21:27 | |
| *** LinuxMe has joined #kata-dev | 21:29 | |
| *** LinuxMe has quit IRC | 21:30 | |
| *** annabelleB has joined #kata-dev | 21:36 | |
| *** eernst has joined #kata-dev | 21:44 | |
| *** dklyle has quit IRC | 21:48 | |
| *** dklyle has joined #kata-dev | 21:50 | |
| *** devimc has joined #kata-dev | 22:05 | |
| *** eernst has quit IRC | 22:11 | |
| *** eernst has joined #kata-dev | 22:13 | |
| *** eernst has quit IRC | 22:18 | |
| *** eernst has joined #kata-dev | 22:18 | |
| *** dlw has joined #kata-dev | 22:21 | |
| *** dlw has quit IRC | 22:41 | |
| *** eernst has quit IRC | 22:47 | |
| *** dlw has joined #kata-dev | 22:50 | |
| *** lamego has left #kata-dev | 23:01 | |
| *** dklyle has quit IRC | 23:01 | |
| *** annabelleB has quit IRC | 23:05 | |
| *** dlw has quit IRC | 23:08 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!