| *** eernst has quit IRC | 00:00 | |
| *** eernst has joined #kata-dev | 00:00 | |
| *** zerocoolback has joined #kata-dev | 00:09 | |
| *** zerocoolback has quit IRC | 00:14 | |
| *** zerocoolback has joined #kata-dev | 00:15 | |
| *** zerocoolback has quit IRC | 00:33 | |
| *** sameo has quit IRC | 01:02 | |
| *** dlw has joined #kata-dev | 01:30 | |
| *** zerocoolback has joined #kata-dev | 02:26 | |
| *** zerocoolback has quit IRC | 02:41 | |
| *** psuriset has quit IRC | 03:01 | |
| *** zerocoolback has joined #kata-dev | 03:22 | |
| *** eernst has quit IRC | 04:21 | |
| *** sjas has joined #kata-dev | 04:22 | |
| *** sjas_ has quit IRC | 04:25 | |
| *** psuriset has joined #kata-dev | 04:25 | |
| *** zerocoolback has quit IRC | 04:54 | |
| *** zerocoolback has joined #kata-dev | 04:55 | |
| *** zerocoolback has quit IRC | 04:58 | |
| *** zerocool_ has joined #kata-dev | 04:58 | |
| *** zerocool_ has quit IRC | 06:07 | |
| *** zerocoolback has joined #kata-dev | 06:07 | |
| *** zerocoolback has quit IRC | 06:08 | |
| *** zerocoolback has joined #kata-dev | 06:09 | |
| *** psuriset_ has joined #kata-dev | 06:27 | |
| *** psuriset__ has joined #kata-dev | 06:30 | |
| *** psuriset has quit IRC | 06:30 | |
| *** psuriset_ has quit IRC | 06:34 | |
| *** psuriset__ has quit IRC | 06:37 | |
| *** zerocoolback has quit IRC | 06:47 | |
| *** jodh has joined #kata-dev | 06:52 | |
| *** dtk has quit IRC | 07:10 | |
| *** dtk has joined #kata-dev | 07:10 | |
| *** zerocoolback has joined #kata-dev | 07:40 | |
| *** davidgiluk has joined #kata-dev | 07:57 | |
| *** gwhaley has joined #kata-dev | 08:09 | |
| *** sameo has joined #kata-dev | 08:09 | |
| *** psuriset__ has joined #kata-dev | 08:23 | |
| kata-irc-bot | <zhangwei555> The CI is still keep failing, I think it should be a CI problem ``` panic: ERROR: docker daemon is not installed, not running, or not accessible to current user: Client: Version: 18.03.1-ce API version: 1.37 Go version: go1.9.5 Git commit: 9ee9f40 Built: Thu Apr 26 07:19:31 2018 OS/Arch: linux/amd64 Experimental: false Orchestrator: swarm Got permission denied while trying to connect to the Docker daemon | 08:38 |
|---|---|---|
| kata-irc-bot | socket at unix:///var/run/docker.sock: Get http://%2Fvar%2Frun%2Fdocker.sock/v1.37/version: dial unix /var/run/docker.sock: connect: permission denied (error exit status 1) goroutine 1 [running, locked to thread]: github.com/kata-containers/runtime/cli.init.2() /tmp/jenkins/workspace/kata-containers-runtime-fedora-27-PR/go/src/github.com/kata-containers/runtime/cli/main_test.go:92 +0xe2c FAILgithub.com/kata-containers/runtime/cli0.097s make: | 08:38 |
| kata-irc-bot | *** [Makefile:386: go-test] Error 1 Build step 'Execute shell' marked build as failure Performing Post build task... Could not match :Build was aborted : False Match found for :Build step 'Execute shell' marked build as failure : True Logical operation result is TRUE Running script : #!/bin/bash ``` Maybe we are not giving enough permission to run the test, so the test program can't visit "/etc/docker/daemon.sock" | 08:38 |
| kata-irc-bot | <zhangwei555> Is anyone from our CI team online? | 08:38 |
| kata-irc-bot | <zhangwei555> See this error from PR: https://github.com/kata-containers/runtime/pull/301 | 08:40 |
| gwhaley | hi @zhangwei555 - so I see some bad docker errors there, and then some odd sudo errors as well. | 08:41 |
| gwhaley | best thing I think is to copy out those errors and past a message on the PR, and copy in fuentess and mnaser | 08:42 |
| gwhaley | looks like some glitch or mis-config with the CI VMs | 08:42 |
| kata-irc-bot | <zhangwei555> @graham.whaley Yes, I believe it should be related to mis-config. Will cc fuentess and @mnaser in PR | 08:44 |
| kata-irc-bot | <zhangwei555> @graham.whaley What's fuentess and mo 's github account? Can't find it | 08:47 |
| gwhaley | @zhangwei555 - thx. I replied and pulled in fuentess (@chavafg) and mnaser :-) They will wake up in 6h or so - fingers crossed! | 09:03 |
| gwhaley | thanks for reporting - yes, we all rely on CI - I know those guys are working hard to get it ironed out... | 09:03 |
| kata-irc-bot | <niteshkonkar007> Can we disable 9pfs and ask kata to use block device instead for rootfs for containers? Is there a document which tells us how to choose between the both. | 09:09 |
| kata-irc-bot | <niteshkonkar007> `#disable_block_device_use = false` and `block_device_driver = "virtio-scsi"` should suffice in `/usr/share/defaults/kata-containers/configuration.toml`? | 09:20 |
| gwhaley | @niteshkonkar007 - yes, I think that is the basis. I think you may also need to use devicemapper as the graph driver, rather than overlay[2] iirc (anybody?). There is a little info in https://github.com/kata-containers/documentation/blob/master/architecture.md#storage | 09:21 |
| gwhaley | buy, yes, it would be great if we had a nice document for that - if you don't want to write it ;-), then could you at least open an issue on kata-containers/documentation repo and /cc amshinde on it pls ? ;-) | 09:22 |
| gwhaley | s/buy/but/ | 09:22 |
| kata-irc-bot | <niteshkonkar007> @graham.whaley: Yes I had read that part. However, I was wondering why its not part of dev-guide, just like we have an option for choosing initrd/rootfs we have option for 9pfs/block_device. Yes I shall open an issue. I am trying non-9pfs way on my x86 block by making changed in the toml file. I am not sure what else I need to do. `graph driver` part I will check as u mentioned. Thnxs | 09:25 |
| kata-irc-bot | <zhangwei555> @niteshkonkar007 If you are using devicemapper as docker's graph driver, I think kata will use block device as default for you | 09:41 |
| *** zerocoolback has quit IRC | 09:55 | |
| kata-irc-bot | <james.o.hunt> @channel - The PR backlog is over 40 again (https://github.com/pulls?utf8=%E2%9C%93&q=is%3Aopen+is%3Apr+org%3Akata-containers). *Please* could folk review some of these, particularly if you are a member of a repo review team. We welcome all review feedback so if you haven't done it before, take a look at https://github.com/kata-containers/community/blob/master/PR-Review-Guide.md. | 10:24 |
| kata-irc-bot | <zhangwei555> And CI is blocking our PRs, we can't make a safe merge without CI's assurance. | 10:33 |
| *** jugs has quit IRC | 10:40 | |
| *** jugs has joined #kata-dev | 10:40 | |
| *** gwhaley has quit IRC | 11:00 | |
| kata-irc-bot | <niteshkonkar007> @zhangwei555: So I changed the config toml file to not use 9pfs + set docker graph driver to devicemapper. I could see that rootfs has been passed as a block device. But I still see 9p in use in `ps -aux | grep qemu` and can also see files like hosts, hostname, resolv.conf being passed as 9pfs. | 11:00 |
| kata-irc-bot | <zhangwei555> Yes, currently 9pfs is still in use for "volume", and hosts, hostname and resolv.conf is passed by docker by default | 11:02 |
| kata-irc-bot | <zhangwei555> So you can't disable 9pfs currently in docker's use case | 11:02 |
| *** jugs has quit IRC | 11:04 | |
| *** jugs has joined #kata-dev | 11:05 | |
| *** cdent has joined #kata-dev | 11:20 | |
| *** dlw1 has joined #kata-dev | 11:32 | |
| *** dlw has quit IRC | 11:34 | |
| *** dlw1 is now known as dlw | 11:34 | |
| *** devimc has joined #kata-dev | 11:53 | |
| *** gwhaley has joined #kata-dev | 12:03 | |
| *** fuentess has joined #kata-dev | 12:22 | |
| *** cdent has quit IRC | 12:29 | |
| *** dlw has quit IRC | 12:43 | |
| *** lamego has joined #kata-dev | 12:51 | |
| *** justJanne has quit IRC | 12:54 | |
| kata-irc-bot | <mnaser> @salvador.fuentes jobs don't run under user 'jenkins' | 12:58 |
| kata-irc-bot | <mnaser> they actually run under the user that the os uses | 12:58 |
| kata-irc-bot | <mnaser> "centos" for centos, "fedora" for fedora, "ubuntu" for ubuntu | 12:58 |
| kata-irc-bot | <mnaser> maybe we need to change the init script to create a jenkins user and copy the ssh keys over to it | 12:58 |
| *** justJanne has joined #kata-dev | 13:00 | |
| kata-irc-bot | <salvador.fuentes> @mnaser, so later on, the whole CI also uses that users for running everything? | 13:01 |
| kata-irc-bot | <mnaser> @salvador.fuentes correct | 13:01 |
| kata-irc-bot | <mnaser> that's why i was using `${USER}` | 13:01 |
| kata-irc-bot | <mnaser> @salvador.fuentes is it the jobs for centos that are failing or all of them? | 13:02 |
| kata-irc-bot | <salvador.fuentes> hmm, then I am not sure why it is failing, it should be transparent to the CI | 13:02 |
| kata-irc-bot | <salvador.fuentes> it is actually failing on all distros | 13:02 |
| kata-irc-bot | <salvador.fuentes> because we need the user (on a unit test) that runs the CI to use docker without sudo | 13:03 |
| kata-irc-bot | <salvador.fuentes> I dont think we have hardcoded the jenkins user somewhere.... | 13:03 |
| kata-irc-bot | <mnaser> i did do that | 13:04 |
| kata-irc-bot | <mnaser> as part of the init it adds that user to the jenkins group | 13:04 |
| kata-irc-bot | <mnaser> i was using `${USER}` but maybe that isnt working | 13:04 |
| kata-irc-bot | <mnaser> let me hardcode it | 13:05 |
| kata-irc-bot | <mnaser> fyi it is here http://jenkins.katacontainers.io/configfiles/ | 13:05 |
| kata-irc-bot | <salvador.fuentes> yes, I actually modified them to also add jenkins user, but as you mention, it does not make sense to do that as we dont have a jenkins user | 13:06 |
| kata-irc-bot | <mnaser> @salvador.fuentes just updated it | 13:06 |
| kata-irc-bot | <mnaser> let me know what you think | 13:06 |
| kata-irc-bot | <salvador.fuentes> taking a look | 13:06 |
| kata-irc-bot | <salvador.fuentes> looks good, lets give it another go to the jobs | 13:07 |
| *** eernst has joined #kata-dev | 13:09 | |
| *** zerocoolback has joined #kata-dev | 13:17 | |
| kata-irc-bot | <eric.ernst> @niteshkonkar007 @zhangwei555 - we were investigating sending these files over gRPC to the agent instead of bind mounting at one point (in case we wanted to completely remove 9p). It'd be a bit of work, and you'd need to monitor the files for changes on the host, since they aren't necessarily static. | 13:29 |
| *** eernst has quit IRC | 13:29 | |
| *** eernst has joined #kata-dev | 13:30 | |
| *** eernst has quit IRC | 13:34 | |
| kata-irc-bot | <zhangwei555> @eric.ernst This is a great idea, I like it. What’s the scenario of modifying network files while container is running? Swarm use case? | 13:43 |
| kata-irc-bot | <salvador.fuentes> @mnaser your changes fixed the issue, thanks, @zhangwei555 @graham.whaley @james.o.hunt the CI should now be ok, let me know if you find something else that blocks you | 14:23 |
| kata-irc-bot | <zhangwei555> Thank you! @salvador.fuentes @mnaser @graham.whaley @james.o.hunt | 14:28 |
| gwhaley | fingers crossed - let's hope we have a stable CI again now :-) | 14:30 |
| *** sameo has quit IRC | 14:50 | |
| *** annabelleB has joined #kata-dev | 15:16 | |
| *** eernst has joined #kata-dev | 15:34 | |
| *** annabelleB has quit IRC | 15:45 | |
| *** lamego1 has joined #kata-dev | 15:53 | |
| *** lamego has quit IRC | 15:56 | |
| *** annabelleB has joined #kata-dev | 15:57 | |
| *** sameo has joined #kata-dev | 16:07 | |
| *** zerocoolback has quit IRC | 16:26 | |
| *** psuriset__ has quit IRC | 16:27 | |
| *** lamego has joined #kata-dev | 16:44 | |
| *** lamego1 has quit IRC | 16:47 | |
| *** jodh has quit IRC | 17:05 | |
| kata-irc-bot | <mnaser> yay awesome re: ci | 17:16 |
| *** gwhaley has quit IRC | 17:22 | |
| kata-irc-bot | <salvador.fuentes> Hi @mnaser, I am debugging an issue that I am seeing a lot in the ubuntu 17.10 VMs. https://github.com/kata-containers/tests/issues/418 | 17:25 |
| kata-irc-bot | <salvador.fuentes> is there a way you can provide me with a 17.10 VM similar to the ones that we use for the CI ? | 17:26 |
| kata-irc-bot | <salvador.fuentes> Tried to reproduce locally, but didnt have any luck | 17:26 |
| kata-irc-bot | <mnaser> @salvador.fuentes public key? | 17:26 |
| kata-irc-bot | <salvador.fuentes> @mnaser: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDA0GRn2YZWsyAfD001DPlvS9sqc2UAS629LFtHtYhVQwYTw4gMzcK+YXcd5uva5Bhiwnb1lOW6o1lLVDESi8T3AUUwzutK4LjWUJjRvop4L2Iquo7iV5HSvKuVysy+tXyzQENeelRUyb3bZKTGes+dGhiGd3AW/tOra1it1BOZEsc9OZImIVqkZSkpboU8ncDSA8rbE+C30bSgnCw9FnNYeC2TFxcXc/L43n9Q5cN2RcqihM1wzaVCTu9xF6f8r9c7LNPvLoDTL9PWHITT8hu8XppMgyPbL8jsuU+0wZ2F4WGIQ65pOxIsW53gI2J8c4jXMqzVD88bS4CNO6ry4D6L fuentess@fuentess-mobl1.amr.corp.intel.com | 17:26 |
| kata-irc-bot | <salvador.fuentes> thanks | 17:26 |
| kata-irc-bot | <mnaser> @salvador.fuentes ubuntu@162.253.55.170 | 17:29 |
| kata-irc-bot | <salvador.fuentes> thank you very much, will let you know when you can destroy it | 17:29 |
| *** annabelleB has quit IRC | 17:50 | |
| *** sameo has quit IRC | 18:02 | |
| *** annabelleB has joined #kata-dev | 18:31 | |
| *** annabelleB has quit IRC | 18:58 | |
| *** eernst has quit IRC | 19:09 | |
| *** eernst has joined #kata-dev | 19:11 | |
| *** davidgiluk has quit IRC | 19:14 | |
| *** eernst has quit IRC | 19:16 | |
| *** eernst has joined #kata-dev | 19:16 | |
| *** eernst has quit IRC | 19:45 | |
| *** eernst has joined #kata-dev | 19:54 | |
| *** eernst has quit IRC | 19:55 | |
| *** eernst has joined #kata-dev | 19:55 | |
| kata-irc-bot | <salvador.fuentes> @mnaser, the VM got frozen, cannot access it anymore. Maybe this is what is happening on the CI jobs. | 19:58 |
| kata-irc-bot | <mnaser> @salvador.fuentes let me look at the console log | 19:59 |
| kata-irc-bot | <mnaser> maybe kernel panic | 19:59 |
| kata-irc-bot | <salvador.fuentes> thanks | 19:59 |
| *** annabelleB has joined #kata-dev | 20:01 | |
| *** annabelleB has quit IRC | 20:04 | |
| *** annabelleB has joined #kata-dev | 20:05 | |
| kata-irc-bot | <mnaser> @salvador.fuentes http://paste.openstack.org/show/723506/ | 20:26 |
| kata-irc-bot | <salvador.fuentes> ups, kernel panic :S | 20:29 |
| kata-irc-bot | <mnaser> @salvador.fuentes is fedora working ok? | 20:30 |
| kata-irc-bot | <salvador.fuentes> yes, fedora is pretty stable | 20:31 |
| kata-irc-bot | <salvador.fuentes> the only one I see unstable is ubuntu 17.10 | 20:31 |
| kata-irc-bot | <mnaser> i wonder if its just a kernel issue | 20:35 |
| kata-irc-bot | <salvador.fuentes> maybe, I think I can try updating the VM and see if it happens again | 20:38 |
| kata-irc-bot | <mnaser> the image should be pretty new but would you like me to rebuild the vm, and then you can apt-get dist-upgrade && reboot ? | 20:42 |
| kata-irc-bot | <salvador.fuentes> yes, please | 20:44 |
| *** eernst has quit IRC | 20:45 | |
| kata-irc-bot | <mnaser> https://git.kernel.org/pub/scm/virt/kvm/kvm.git/commit/?id=8eb3f87d903168bdbd1222776a6b1e281f50513e | 20:47 |
| kata-irc-bot | <mnaser> looks like that might be a cause but we'll see | 20:47 |
| *** eernst has joined #kata-dev | 20:47 | |
| *** eernst has quit IRC | 20:47 | |
| *** eernst has joined #kata-dev | 20:47 | |
| kata-irc-bot | <mnaser> @salvador.fuentes rebuilt, please let me know if updating lands you a newer kernel... | 20:50 |
| *** devimc has quit IRC | 20:52 | |
| kata-irc-bot | <salvador.fuentes> thanks, but as you said, there is no new kernel available | 20:59 |
| *** eernst has quit IRC | 21:00 | |
| *** eernst has joined #kata-dev | 21:00 | |
| *** eernst has quit IRC | 21:02 | |
| kata-irc-bot | <salvador.fuentes> I am comparing the kernel version that the azure machines provide, but it is the same. Thought there might be a difference, since I couldn't reproduce the issue there | 21:04 |
| *** annabelleB has quit IRC | 21:04 | |
| *** eernst has joined #kata-dev | 21:06 | |
| *** eernst has quit IRC | 21:10 | |
| *** annabelleB has joined #kata-dev | 21:11 | |
| *** eernst has joined #kata-dev | 21:23 | |
| *** eernst has quit IRC | 21:27 | |
| *** eernst has joined #kata-dev | 21:31 | |
| *** eernst has quit IRC | 21:35 | |
| *** eernst has joined #kata-dev | 21:37 | |
| *** annabelleB has quit IRC | 21:40 | |
| *** eernst has quit IRC | 21:42 | |
| *** eernst has joined #kata-dev | 21:43 | |
| *** eernst has quit IRC | 21:48 | |
| kata-irc-bot | <salvador.fuentes> @mnaser, you can delete the VM, it is already hanged | 21:48 |
| *** fuentess has quit IRC | 21:48 | |
| *** eernst has joined #kata-dev | 21:49 | |
| *** eernst has quit IRC | 21:51 | |
| *** eernst has joined #kata-dev | 21:53 | |
| *** eernst has quit IRC | 21:55 | |
| *** eernst has joined #kata-dev | 21:55 | |
| kata-irc-bot | <mnaser> aw damn | 22:17 |
| kata-irc-bot | <mnaser> that's annoying | 22:17 |
| kata-irc-bot | <mnaser> and it didn't hang running on azure? | 22:17 |
| kata-irc-bot | <mnaser> @salvador.fuentes i have a redhat kvm person who i can refer this too.. is it possible for you to make this down to small reproducer? | 22:18 |
| *** annabelleB has joined #kata-dev | 22:41 | |
| *** dklyle has quit IRC | 22:56 | |
| *** dklyle has joined #kata-dev | 22:57 | |
| *** lamego has quit IRC | 23:16 | |
| kata-irc-bot | <salvador.fuentes> @mnaser I think I can try to make a script that installs kata-containers and launch containers using docker, I think we could reproduce it | 23:28 |
| kata-irc-bot | <salvador.fuentes> I'll make the script and let you know when I have it. will ask you again for another machine and see if that reproduces the issue | 23:29 |
| *** zerocoolback has joined #kata-dev | 23:35 | |
| *** zerocoolback has quit IRC | 23:38 | |
| kata-irc-bot | <mnaser> @salvador.fuentes thanks. That would be a win for everyone. A very simple small reproducer as t seems to happen all the time | 23:38 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!