Tuesday, 2018-07-03

« Monday, 2018-07-02 Index Wednesday, 2018-07-04 »
*** zerocoolback has joined #kata-dev00:18
*** zerocoolback has quit IRC00:34
clarkbsebastien.boeuf ok should be up and running again. Looks like k8s failed though http://logs.openstack.org/74/74/c3a54a71613ec4736adcdb363be6c17b01ccd9f1/third-party-check/kata-runsh/4fee03e/job-output.txt.gz but crio passed00:37
*** zerocoolback has joined #kata-dev00:43
*** zerocoolback has quit IRC00:50
*** dklyle has quit IRC00:55
*** yingjun has joined #kata-dev01:04
*** zerocoolback has joined #kata-dev02:27
*** yingjun has quit IRC02:31
*** yingjun has joined #kata-dev02:43
*** dlw has joined #kata-dev02:54
*** yingjun has quit IRC03:09
*** zerocoolback has quit IRC03:09
*** yingjun has joined #kata-dev03:15
*** yingjun has quit IRC03:23
*** zerocoolback has joined #kata-dev03:24
kata-irc-bot1<xu> @jose.carlos.venegas.m @sebastien.boeuf what03:37
kata-irc-bot1<xu> @jose.carlos.venegas.m @sebastien.boeuf what’s the status of issue runtime#383 (vsocks support)? If there has no works done, we could help to push it.03:39
kata-irc-bot1<harshal.patil> @bergwolf I am wondering how can we make generic storage struct at runtime for the suggestion that @zhangwei555 is making here, https://github.com/kata-containers/runtime/issues/443#issuecomment-40020057603:45
kata-irc-bot1<harshal.patil> I will try something out, let me know if that's what you would intended. I will update the PR and tag you for comments.03:46
kata-irc-bot1<bergwolf> @harshal.patil you can create a sandbox level storage data structure in kata agent, and use it to do reference counting to allow the storage to be shared across containers03:48
kata-irc-bot1<bergwolf> sure, ping me when you update the PR. thanks!03:48
kata-irc-bot1<harshal.patil> I am getting confused by this term `reference counting`. What does it mean? Any examples of that in kata right now?03:49
kata-irc-bot1<harshal.patil> why do we need it?03:49
kata-irc-bot1<harshal.patil> sure03:51
*** dlw has quit IRC04:02
*** dlw has joined #kata-dev04:03
kata-irc-bot1<zhangwei555> We can discuss more publicly in issue https://github.com/kata-containers/runtime/issues/443#issuecomment-400200576  @harshal.patil04:12
kata-irc-bot1<harshal.patil> sure04:12
kata-irc-bot1<harshal.patil> so you suggest I wait before we hear from others?04:13
kata-irc-bot1<harshal.patil> @zhangwei55504:13
kata-irc-bot1<harshal.patil> I almost started coding for what you had suggested.04:13
*** sjas_ has joined #kata-dev04:25
*** sjas has quit IRC04:28
*** LinuxMe has joined #kata-dev04:33
*** LinuxMe has quit IRC04:37
*** zerocoolback has quit IRC05:15
*** zerocoolback has joined #kata-dev05:15
*** zerocoolback has quit IRC05:15
*** zerocoolback has joined #kata-dev05:16
*** zerocoolback has quit IRC05:16
*** zerocoolback has joined #kata-dev05:17
*** zerocoolback has joined #kata-dev05:17
*** zerocoolback has quit IRC05:18
*** zerocoolback has joined #kata-dev05:18
kata-irc-bot1<zhangwei555> I think so, we can wait for other's opinions05:24
*** sameo has quit IRC06:12
*** jodh has joined #kata-dev06:57
*** jodh has quit IRC06:58
*** jodh has joined #kata-dev06:58
*** sameo has joined #kata-dev07:11
*** LinuxMe has joined #kata-dev07:14
*** LinuxMe has quit IRC07:19
kata-irc-bot1<linxiulei> Hi guys, did you have demands for io limits in kata containers? And I saw process vhost-net consume lots of cpus as well07:20
*** LinuxMe has joined #kata-dev07:52
*** LinuxMe has quit IRC07:57
*** davidgiluk has joined #kata-dev08:04
*** gwhaley has joined #kata-dev08:05
*** LinuxMe has joined #kata-dev08:28
*** LinuxMe has quit IRC08:32
*** cdent has joined #kata-dev08:56
*** cdent_ has joined #kata-dev09:04
*** cdent has quit IRC09:05
*** cdent_ is now known as cdent09:05
*** zerocoolback has joined #kata-dev09:20
kata-irc-bot1<harshal.patil> @bergwolf @zhangwei555 https://github.com/kata-containers/runtime/pull/45809:20
*** LinuxMe has joined #kata-dev09:21
*** LinuxMe has quit IRC09:23
*** zerocoolback has quit IRC09:24
*** sjas_ is now known as sjas09:32
*** zerocoolback has joined #kata-dev09:46
*** zerocoolback has quit IRC09:51
*** cdent has quit IRC10:25
*** sameo has quit IRC10:43
*** dlw has quit IRC10:50
*** yingjun has joined #kata-dev11:42
*** devimc has joined #kata-dev11:51
*** yingjun has quit IRC11:52
*** zerocoolback has joined #kata-dev12:02
*** LinuxMe has joined #kata-dev12:03
*** LinuxMe has quit IRC12:07
*** zerocoolback has quit IRC12:32
*** zerocoolback has joined #kata-dev12:32
*** zerocoolback has quit IRC12:34
*** zerocoolback has joined #kata-dev12:37
*** LinuxMe has joined #kata-dev12:37
*** zerocoolback has quit IRC12:37
*** zerocoolback has joined #kata-dev12:38
*** fuentess has joined #kata-dev12:41
*** fuentess has quit IRC12:42
*** fuentess has joined #kata-dev12:42
*** LinuxMe has quit IRC12:42
*** zerocoolback has quit IRC12:42
*** LinuxMe has joined #kata-dev12:44
*** zerocoolback has joined #kata-dev12:45
*** zerocoolback has quit IRC12:47
*** zerocoolback has joined #kata-dev12:56
*** zerocoolback has quit IRC12:56
*** zerocoolback has joined #kata-dev12:57
*** zerocoolback has quit IRC13:01
*** zerocoolback has joined #kata-dev13:01
*** zerocoolback has quit IRC13:02
*** zerocoolback has joined #kata-dev13:03
*** zerocoolback has quit IRC13:04
*** zerocool_ has joined #kata-dev13:10
*** zerocoolback has joined #kata-dev13:11
*** lamego has joined #kata-dev13:34
*** yingjun has joined #kata-dev14:08
*** annabelleB has joined #kata-dev14:09
kata-irc-bot1<julio.montes> Hi @anne, did you receive an e-mail from canonical ?14:14
kata-irc-bot1<anne> @julio.montes nothing in my inbox, but let me go through zendesk spam again14:15
kata-irc-bot1<julio.montes> haha, thanks14:15
kata-irc-bot1<anne> @julio.montes sure enough. Let me forward this to you14:18
kata-irc-bot1<julio.montes> @anne Thanks :slightly_smiling_face:14:18
kata-irc-bot1<anne> @julio.montes oh, wait. This is just a form marketing email waiting us to take one of their tutorials. I thought our (human) needed help with a yaml file. (I haven't had coffee yet)14:19
kata-irc-bot1<julio.montes> ohh ok14:20
kata-irc-bot1<anne> no actual emails yet :(14:21
*** jodh has quit IRC14:47
*** jodh has joined #kata-dev14:49
*** jodh has quit IRC14:49
*** jodh has joined #kata-dev14:49
*** zerocoolback has quit IRC14:56
*** jodh has quit IRC14:57
*** yingjun has quit IRC15:23
kata-irc-bot1<salvador.fuentes> clarkb: hi, @sebastien.boeuf and I were checking the latest log you pasted yesterday and it seems that the k8s issue was already fixed yesterday morning, I was wondering if you are  pulling latest version of the tests repository15:36
clarkbyes, it should pull master. Let me see if it logged the sha1 it used.15:37
clarkbsalvador.fuentes: it pulls it via the .ci/setup.sh script in the proxy repo I think15:37
kata-irc-bot1<salvador.fuentes> clarkb: yes, it should pull latest changes on tests repo15:38
kata-irc-bot1<salvador.fuentes> I'll run a recheck and see if the issue persists15:38
clarkbhttp://logs.openstack.org/74/74/c3a54a71613ec4736adcdb363be6c17b01ccd9f1/third-party-check/kata-runsh/4fee03e/job-output.txt.gz#_2018-07-02_23_59_10_626994 is all it logs I think15:39
clarkbalso fedora is still failing the go test suite http://logs.openstack.org/74/74/c3a54a71613ec4736adcdb363be6c17b01ccd9f1/third-party-check/kata-runsh-fedora-28/1183881/job-output.txt.gz#_2018-07-03_00_26_10_529817 not crashing entirely though which is good. Are you still using the test node that I held for you?15:40
*** zerocoolback has joined #kata-dev15:41
*** dklyle has joined #kata-dev15:41
kata-irc-bot1<salvador.fuentes> hmmm, again the fdisk: command not found, which I couldnt reproduce when running the tests15:42
clarkband I double checked fdisk is there. Maybe it is a path issue?15:42
*** zerocoolback has quit IRC15:42
kata-irc-bot1<salvador.fuentes> clarkb, no, I am not using it anymore, you can delete it, I'll try to check the issues on a machine I have15:42
*** zerocoolback has joined #kata-dev15:43
kata-irc-bot1<salvador.fuentes> yeah, maybe a PATH issue,15:43
clarkbI'll get that test node we held cleaned up, thank you for confirming you are done with it15:43
*** zerocoolback has quit IRC15:47
kata-irc-bot1<salvador.fuentes> thank you15:53
clarkbsalvador.fuentes: http://logs.openstack.org/74/74/c3a54a71613ec4736adcdb363be6c17b01ccd9f1/third-party-check/kata-runsh/b570751/job-output.txt.gz#_2018-07-03_16_16_35_278773 seems to still be failing on k8s.16:40
kata-irc-bot1<salvador.fuentes> taking a look16:42
kata-irc-bot1<salvador.fuentes> clarkb: do we have access to the kubelet journal from that execution?16:42
clarkbhttp://logs.openstack.org/74/74/c3a54a71613ec4736adcdb363be6c17b01ccd9f1/third-party-check/kata-runsh/b570751/logs/kubelet.log.gz that one?16:43
kata-irc-bot1<salvador.fuentes> thanks16:43
clarkbI tried to grab all the logs that the jenkins jobs were grabbing :)16:43
kata-irc-bot1<salvador.fuentes> clarkb: is there a chance I can get access into that machine? I was taking a look at the kubelet log, but cannot get what get wrong16:52
clarkbthat machine will already have been deleted, but I can set up a hold for the next run16:52
kata-irc-bot1<salvador.fuentes> clarkb: yes please16:59
clarkbsalvador.fuentes https://zuul.openstack.org/stream.html?uuid=5467bd1870d349a49e8673690a4901e8&logfile=console.log is the next one which should be held. Currently running though, you should wait for the job to complete before making changes (otherwise you could impact the result). root@162.253.55.217:07
kata-irc-bot1<salvador.fuentes> ok, thanks clarkb, will wait until it fails17:08
*** gwhaley has quit IRC17:13
*** ObiWanKentNobi has joined #kata-dev17:15
*** annabelleB has quit IRC17:27
*** annabelleB has joined #kata-dev17:28
clarkbsalvador.fuentes seems to be done now17:46
clarkband it failed in the same manner17:46
kata-irc-bot1<salvador.fuentes> clarkb, I'm already inside the machine, lets see why it fails17:48
clarkbI've got to pop out for a few and prep for a meeting but feel free to ping me if you have questions and I will do my best17:52
*** dklyle has quit IRC17:53
*** ObiWanKentNobi has quit IRC18:10
*** cdent has joined #kata-dev18:13
*** davidgiluk has quit IRC19:03
*** dgonzalez has quit IRC19:14
*** dgonzalez has joined #kata-dev19:16
*** dgonzalez has quit IRC19:21
*** dgonzalez has joined #kata-dev19:26
*** LinuxMe has quit IRC19:29
*** LinuxMe has joined #kata-dev19:40
*** annabelleB has quit IRC20:29
kata-irc-bot1<sebastien.boeuf> clarkb: hey, quick question, is there any way to disable ipv6 on those vexxhost machines ? Just to make sure we don't have some issues in Kata related to this20:49
*** annabelleB has joined #kata-dev20:49
*** devimc has quit IRC20:50
clarkbyou can delete the ip address from the interface20:50
clarkbwhich should remove routes for it as well20:51
clarkb(though vexxhost does RAs so you may have to disable listening for those too)20:51
clarkbif we can show that is the problem then we can figure out if it is approprite to stop using ipv620:52
clarkb(in general though we think it is a good idea specifically for these reasons)20:52
kata-irc-bot1<archana.m.shinde> @sebastien.boeuf echo " net.ipv6.conf.all.disable_ipv6=1 net.ipv6.conf.default.disable_ipv6=1 net.ipv6.conf.lo.disable_ipv6=1 " | sudo tee -a /etc/sysctl.conf21:00
kata-irc-bot1<archana.m.shinde> that should disable ipv6 completely21:00
kata-irc-bot1<sebastien.boeuf> @archana.m.shinde thx ! how do I make sure this is taken into account ?21:03
kata-irc-bot1<sebastien.boeuf> any service I have to restart ?21:03
clarkbyou can use the sysctl command to set them right away (there is probably a systemd unit that will reread the /etc config too)21:04
clarkbthen check ip addr output for ipv6 addresses21:04
kata-irc-bot1<archana.m.shinde> @sebastien.boeuf "sysctl -p" should do the trick21:07
kata-irc-bot1<sebastien.boeuf> yes I found the same thanks !21:08
kata-irc-bot1<archana.m.shinde> @sebastien.boeuf you are seeing issues with k8s?21:09
*** isaagar has quit IRC21:11
kata-irc-bot1<sebastien.boeuf> in case of Zuul, the kube-dns (being a Kata pod), cannot access 10.96.0.1 which is the apiserver service21:11
kata-irc-bot1<archana.m.shinde> oh ok21:14
clarkbthat should be independent of ipv6 as we run dual stack21:14
kata-irc-bot1<archana.m.shinde> yeah we ignore ipv6 in Kata21:14
kata-irc-bot1<archana.m.shinde> wonder if ipv6 can be disabled with k8s21:15
clarkbk8s doesn't support ipv6 last I checked21:15
clarkbso everything should just ignore ipv6 I would think21:15
kata-irc-bot1<salvador.fuentes> clarkb: do you know if the machines that we use on jenkins have the same network configuration as the one that you launch using zuul?21:15
kata-irc-bot1<salvador.fuentes> I ask because we are not facing these issues in the jenkins jobs21:16
clarkbI don't, we use vexxhost's default networking which includes ipv4 and ipv6. I don't know if the jenkins jobs do somethingdifferent21:17
clarkbI'm looking at the test node we held earlier and I don't see any of the 10 net networking there? Am I just failing at looking?21:17
kata-irc-bot1<archana.m.shinde> yeah seeing flannel does not support ipv6 yet21:17
clarkbdocker is on 172.17.0.1/1621:19
clarkbhost interface is 162.253.55.2/24. Where does 10.96.0.1 come from? or is that interface only there when k8s is running?21:20
kata-irc-bot1<salvador.fuentes> yes, that comes from k8s21:21
kata-irc-bot1<salvador.fuentes> need to go offline now, I'll try to connect later21:22
kata-irc-bot1<archana.m.shinde> @salvador.fuentes @sebastien.boeuf Does kubectl logs {podid} give any more details?21:23
clarkbmy hunch is actually iptables21:27
clarkbwe firewall test instances off to protect themselves from each other (and prevent things like open resolvers from being on the Internet)21:27
clarkbThere is an iptables rule that kubernetes seems to set to allow traffic to the dns server21:29
clarkbbut I don't see anything like that for the api server on first glance21:29
clarkboh except those are rejects21:30
*** isaagar has joined #kata-dev21:31
clarkbsalvador.fuentes: yes, that is my hunch. I see rules for the --cluster-cidr range but not the --service-cluster-ip-range range21:35
*** LinuxMe has quit IRC21:45
*** annabelleB has quit IRC22:01
*** LinuxMe has joined #kata-dev22:17
*** isaagar has quit IRC22:17
*** LinuxMe has quit IRC22:21
*** cdent has quit IRC22:23
clarkbsalvador.fuentes: Jul 03 22:32:08 ubuntu-xenial-vexxhost-vexxhost-ca-ymq-1-0000521928 kernel: iptables dropped: IN=cni0 OUT= PHYSIN=veth44664e1c MAC=0a:58:0a:f4:00:01:0a:58:0a:f4:00:02:08:00 SRC=10.244.0.2 DST=162.253.55.2 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=50193 DF PROTO=TCP SPT=49334 DPT=6443 WINDOW=28200 RES=0x00 SYN URGP=0 that seems to confirm it. Its actually hitting 10.96.0.1 via the NAT from the22:33
clarkbmain ip:644322:33
clarkbI think we need to add a rule that allows --cluster-cidr range to talk to the main host ip22:33
*** LinuxMe has joined #kata-dev23:05
*** LinuxMe has quit IRC23:09
« Monday, 2018-07-02 Index Wednesday, 2018-07-04 »

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!