Friday, 2017-03-24

*** caoyuan has joined #openstack-kolla00:06
*** adrian_otto has quit IRC00:16
*** yangyapeng has joined #openstack-kolla00:19
*** yangyapeng has quit IRC00:20
kolla-slack<kfox1111> @fooliouno setting to different nodes is tested in the gate. works good. :slightly_smiling_face:00:27
*** caoyuan has quit IRC00:29
*** dave-mccowan has quit IRC00:29
openstackgerritMohammed Naser proposed openstack/kolla master: Merged multiple ENV into a single layer  https://review.openstack.org/44937100:30
*** jrobinson has quit IRC00:31
*** duonghq has joined #openstack-kolla00:34
duonghqmorning guys00:34
*** sayantani01 has quit IRC00:43
*** dave-mccowan has joined #openstack-kolla00:49
spsuryaduonghq: morning00:54
duonghqmorning spsurya00:54
*** dave-mcc_ has joined #openstack-kolla00:55
duonghqsdake, ping00:57
*** dave-mccowan has quit IRC00:57
*** tovin07 has joined #openstack-kolla00:58
*** qiliang27 has joined #openstack-kolla01:05
*** jrobinson has joined #openstack-kolla01:08
*** dave-mccowan has joined #openstack-kolla01:09
*** dave-mcc_ has quit IRC01:11
*** qiliang28 has joined #openstack-kolla01:12
*** sayantani01 has joined #openstack-kolla01:13
*** qiliang27 has quit IRC01:14
*** qiliang28 is now known as qiliang2701:14
*** egonzalez has quit IRC01:15
*** MasterOfBugs has joined #openstack-kolla01:17
duonghqsbezverk_, are you there?01:22
*** zhurong has joined #openstack-kolla01:26
openstackgerrityunfeng zhou proposed openstack/kolla-ansible master: change log-* to flog-*.  https://review.openstack.org/44938401:26
openstackgerrityunfeng zhou proposed openstack/kolla-ansible master: change log-* to flog-*.  https://review.openstack.org/44938401:27
*** cuongnv has joined #openstack-kolla01:28
*** zhubingbing has joined #openstack-kolla01:28
kolla-slack<portdirect> Oh this is cool o/ hey peeps :D01:35
*** japestinho has quit IRC01:46
*** japestinho has joined #openstack-kolla01:47
openstackgerritjimmygc proposed openstack/kolla master: Add vSphere Support  https://review.openstack.org/44939502:02
openstackgerritjimmygc proposed openstack/kolla master: Add vSphere Support  https://review.openstack.org/44939502:10
*** eaguilar has joined #openstack-kolla02:13
duonghqJeffrey4l, are you there?02:15
Jeffrey4lduonghq, yep. sup02:17
duonghqhttps://review.openstack.org/#/c/445573/ about this ps, where do you get the cherry pick commit id?02:18
Jeffrey4linc0, please review the release patch https://review.openstack.org/448711 ?02:18
duonghqthe commit id I get from cherry-pick feature of gerrit is differ from your02:18
Jeffrey4lduonghq, check https://review.openstack.org/#/q/I6b77773238cd65a3cc009d8d9a38bdbea0035212,n,z02:18
Jeffrey4lduonghq, the commit sha will be changed, for sure.02:19
Jeffrey4lit is how git works.02:19
kolla-slack<inc0> Done Jeffrey02:20
duonghqhmm, given that gerrit can search for this02:20
duonghqI think it's ok02:20
Jeffrey4linc0, cool. whats this? slack channel?02:20
duonghqalthough I prefer merged id02:21
kolla-slack<inc0> Yeah sdake connected slack with irc02:21
Jeffrey4lduonghq, yep. as long as the change id is the same.02:21
kolla-slack<inc0> Writing from phone02:21
Jeffrey4linc0, how can i join it?02:21
kolla-slack<inc0> Sure, Kubernetes slack team, kolla02:22
kolla-slack<inc0> It's open channel02:22
duonghqhmm, we have this channel long ago02:22
duonghqbut kolla-slack is new?02:22
kolla-slack<inc0> No, it's same channel02:22
kolla-slack<inc0> Just now it's mirrored02:22
*** caoyuan has joined #openstack-kolla02:23
*** caowei has quit IRC02:23
duonghqI mean the kolla-slack bot is new?02:23
duonghqhave not seen it before02:24
*** yangyapeng has joined #openstack-kolla02:25
*** sayantani01 has quit IRC02:27
*** schwicht has quit IRC02:27
kolla-slack<jeffrey4l> i am in. interesting02:27
duonghqJeffrey4l, have fun ;)02:28
*** caoyuan has quit IRC02:29
*** caoyuan has joined #openstack-kolla02:31
openstackgerritjimmygc proposed openstack/kolla master: Add vSphere Support  https://review.openstack.org/44939502:33
sdakesup peeps02:44
sdakeduonghq kolla-aslack is a bot i setup02:44
sdakeduonghq it is new indeed02:45
duonghqnice02:45
*** unicell1 has quit IRC02:45
sdakekolla-slack isa a bot that runs on a server and connects to slack and irc and mirrors the conversation02:46
*** shashank_t_ has joined #openstack-kolla02:46
sdakeit took about 30 minutes to tsetup up02:46
duonghqdoes it consume much resource?02:47
sdakewe couldn't set it up prior02:47
sdakethe resource it consumes the msot of is the scrollback (in the free lack)02:47
sdakeI think someone in kubernetes land is paying for paid slack which is 7 or 8 bucks a month per user02:47
sdakeweh ave between 800-1k messages a day02:48
Jeffrey4lsdake, set up the bot from zero? or leverage some existing service?02:48
sdakefree slack has a 10k msg scrollback before messages drop off in free02:48
sdakeJeffrey4l i used this : https://github.com/ekmartin/slack-irc02:48
sdakenpm install slack-irc02:49
sdakeand sepnd 3 minutes wirting a cnofig file02:49
sdakeGTG02:49
Jeffrey4lroger. thanks.02:49
sdakethatis good to go02:49
sdakethere is a paid service that does the same thing which is 490$/yr02:49
* sdake starts a business hosting slack-irc instances :)02:50
Jeffrey4llol02:50
Jeffrey4lbtw, slack is hard to use in China. very slow in sometime. ;(02:50
sdakeJeffrey4l right - plan is to still continue to use irc02:50
sdakehwoever there are alot of people that either don't want or can't run irc for whatever eason02:51
Jeffrey4lsync bot is a good idea.02:51
sdakebut can run slack - so its essentiallly free for others to participate in kolla dev02:51
spsuryasdake: +102:53
spsuryasdake: my team is also using slack02:53
spsuryagood to use02:53
sdakethe slack side integration is pretty good, the irc side integration isa little clunky02:57
sdakei think we mostly want to focus our work here - but provide an opportunity to build a bridge to the kubernetes community - and part of that happens via slack :)02:58
sdakehere being irc02:58
*** yuanying has quit IRC02:58
*** eaguilar has quit IRC03:04
openstackgerritjimmygc proposed openstack/kolla master: Add vSphere Support  https://review.openstack.org/44939503:08
*** dave-mccowan has quit IRC03:08
spsuryaJeffrey4l:  sdake btw what is the problem with IRC ?03:26
Jeffrey4li do not see the issue of irc. but slack is easy to integrate with other service. and k8s team use slack and openstack communit use irc03:26
sdakeright the problem we are fixing is not that irc is broken but that the communities don't have a common way to communicate that doesn't involve two clients03:28
sdakenow they do :)03:28
sdakethey/we03:28
spsuryaJeffrey4l: sdake roger03:30
openstackgerritjimmygc proposed openstack/kolla master: Add vSphere Support  https://review.openstack.org/44939503:32
Jeffrey4lbtw, i found http://wiki.znc.in/ZNC  . it is a IRC bouncer03:33
*** Margin has joined #openstack-kolla03:34
spsuryaJeffrey4l: if we will integrate slack then we will have the same problem for kolla developer and other service developer in  communication03:35
spsuryasdake03:35
Jeffrey4lwhat's the "same problem"?03:36
sdakemust have missed a message03:37
sdake"same problem?"03:37
spsuryaJeffrey4l: as we have in communication with kubernetes member as i understand :(03:38
sdakespsurya if you want to talk directly with helm community or kubernetes community - the correct answer is to use the tools they are accustomed to03:39
sdake(which means slack)03:39
sdakewe can't necessarily solve that problem03:39
Jeffrey4li do not think we have better solution expect the current sync bot.03:39
sdakethe problem we can solve is linking hte kubernetes slack #kolla to #openstack-kolla on irc03:40
sdakespsurya the kubernetes slack is open to anyone03:40
sdakespsurya not sure what the exact issue you are concerned with03:40
sdakespsurya mind expanding03:40
spsuryaJeffrey4l:  sdake roger03:42
spsuryasdake: as i understand the problem is linking the kuberenes slack and #openstack-kolla03:46
spsuryaright ?03:46
spsuryai.e running common channel of slack and IRC03:47
spsuryafor kolla-kuberenetes03:47
spsuryaand kubernetes03:47
spsuryasdake: i have no issue in communication with helm community or kubernetes  community :)03:50
spsuryajust wanted to understand the yours and Jeffrey4l conversation03:50
spsuryaJeffrey4l: sdake BRB03:52
*** unicell has joined #openstack-kolla03:53
spsuryasorry for too much mind expansion :(03:53
*** cliles has quit IRC04:01
*** zhurong has quit IRC04:07
*** n0isyn0ise has quit IRC04:12
*** n0isyn0ise has joined #openstack-kolla04:14
*** unicell1 has joined #openstack-kolla04:18
*** unicell has quit IRC04:18
*** jrobinson has quit IRC04:19
*** yuanying has joined #openstack-kolla04:20
*** unicell has joined #openstack-kolla04:22
*** unicell1 has quit IRC04:23
*** jrobinson has joined #openstack-kolla04:24
openstackgerritMichal Jastrzebski (inc0) proposed openstack/kolla master: DO NOT MERGE: PoC of CI friendly share nothing img  https://review.openstack.org/44942604:33
*** caoyuan has quit IRC04:35
*** yangyapeng has quit IRC04:40
sdakenothing like the "please stop" passive aggressive response in a review04:43
* sdake cackles insanely04:43
*** skramaja has joined #openstack-kolla04:44
duonghqsdake, sup?04:47
sdakeduonghq nothing much04:48
sdakeduonghq just personal attacks in reviews followed by "please stop"04:48
duonghqya04:48
duonghqnot good04:48
sdakei dont understand why people can't just accept there is a system in place to handle these things and work through the normal processes and channels to make things work well04:48
sdakeinstead attack me for stating facts04:49
sdakeits not like I wrote linuxfoundation's trademark policy04:49
sdakeI simply pointed it out04:49
duonghqhmm,04:50
sdakehttps://review.openstack.org/44942604:50
sdakefor those interested  - bring some salt !04:50
duonghqI don't get idea behind it04:51
sdakewho knows who cares04:51
duonghqah, got the idea04:51
sdakeif someone wants to do something wfm04:51
duonghqone of the PTG topic04:51
sdakeyup that may be it04:52
duonghqhmm, don't understand why keystone do not get resolve.conf from configmap04:53
sdakesbezverk_ speaking of resolv.conf04:53
duonghqit was ok with kargo04:53
sdakesbezverk_ that workaround you ahve in place is not needed if docker-engine is setup properly04:53
duonghqbut when I setup cluster with kubeadm, it doesn't like me anymore04:53
*** unicell1 has joined #openstack-kolla04:53
sdakekubeadm uses a resolv.conf with the name "kube-dns"04:54
sdakekargo uses "kubedns"04:54
sdakethis is a delta04:54
duonghqya, I fix that04:54
duonghqfixed04:54
*** unicell has quit IRC04:54
duonghqthe generated configmap is ok04:55
*** haplo37 has quit IRC04:59
*** haplo37 has joined #openstack-kolla05:08
Jeffrey4lspsurya, around? wanna talk about https://review.openstack.org/432300  this patch with u.05:09
spsuryaJeffrey4l: yes05:10
Jeffrey4lspsurya, still can not know why we need the old version.05:10
Jeffrey4lwhat the new and old is used for the same purpose.  and there are some issue in the old implementation.05:11
Jeffrey4lcan not parse what u said " Supose if any operator wants to find the latest version of kolla rather than version of the latest version from openstack release"05:12
Jeffrey4lspsurya, ^^05:12
spsuryaJeffrey4l: new implementation is good for getting the latest version from openstack/release05:12
spsuryaJeffrey4l: i want to say that that suppose if we want to get the version for specific release05:13
Jeffrey4lfor example?05:13
Jeffrey4lopenstack release? or nova release?05:13
spsuryaJeffrey4l: I mean for mitaka, Newton or ocata05:14
spsuryanot the latest i.e ocata05:14
spsuryaJeffrey4l: new implemenattion is good05:15
Jeffrey4lspsurya, the old implement can not handle this but the new implement can handle this.05:15
Jeffrey4lthis is what i said: old have some bug.05:15
Jeffrey4lold implement always get the latest version.05:15
Jeffrey4lhe do not know which version is the latest ocata version.05:15
Jeffrey4lthen it resulted in we are using newton swift in mitaka branch, now.05:16
*** zhurong has joined #openstack-kolla05:16
Jeffrey4lspsurya, check the --openstack-release parameter in the new implementation.05:16
spsuryai Jeffrey4l sorry but if we see from commit message it look like what i am saying05:16
Jeffrey4land it use .gitreview branch as default.05:17
spsuryaJeffrey4l: roger05:17
Jeffrey4lversion-check gets latest versions from05:17
Jeffrey4lopenstack/releases repo, which is faster and get also the latest05:17
Jeffrey4lversions for certain openstack release.05:17
spsuryaJeffrey4l: checking05:17
Jeffrey4li.e.: it support master and certain openstack release.05:17
spsuryaJeffrey4l: I also commented *Please correct if I am on wrong track. Thanks !!*05:18
spsuryaJeffrey4l:  i think we need to correct  the commit message05:19
Jeffrey4lyep. so we are talking this ;)05:19
Jeffrey4lyep. the commit message is a little confused .05:19
spsuryaJeffrey4l: that is why i got confused05:19
Jeffrey4las a non-native speaker, this is always hard for me ;)05:19
spsuryaJeffrey4l: updation of commit message will make me green for you patch :)05:20
Jeffrey4li will add some usage in the commit message for clarify05:20
Jeffrey4lcool.05:20
spsuryaJeffrey4l: roger , thanks05:20
Jeffrey4lthanks.05:20
spsuryaJeffrey4l: any way !  what about the your travellng to summit ?05:21
spsuryaJeffrey4l: Are we going to be on same stage ?05:21
spsurya:)05:21
Jeffrey4lstill preparing the visa. others are find. if it will be accepted.05:22
Jeffrey4lyep. hope that so.05:22
Jeffrey4lwhat's your progress?05:22
spsuryasame in VISA process05:22
spsurya:(05:22
Jeffrey4laha05:22
spsuryaJeffrey4l: hope we will get that05:24
openstackgerritJeffrey Zhang proposed openstack/kolla master: Get OpenStack service versions from openstack/releases repo  https://review.openstack.org/43230005:26
Jeffrey4lspsurya, updated. please check again. ^^05:27
*** caoyuan has joined #openstack-kolla05:27
spsuryaJeffrey4l: sure05:27
Jeffrey4lthanks.05:29
spsuryaJeffrey4l: LGTM05:38
Jeffrey4lthanks.05:38
spsuryawaiting for jenkin jobs05:39
*** sayantani01 has joined #openstack-kolla05:45
*** bmace has quit IRC05:48
*** bmace has joined #openstack-kolla05:48
*** shashank_t_ has quit IRC05:53
*** mdnadeem has joined #openstack-kolla05:54
*** shashank_t_ has joined #openstack-kolla05:59
duonghqanybody get TSP result?06:02
*** shashank_t_ has quit IRC06:03
duonghqspsurya, did you get TSP result?06:05
spsuryaduonghq: didn't apply, hoping from employer :)06:06
duonghqok06:06
*** yee37914 has quit IRC06:06
*** yee379 has joined #openstack-kolla06:08
*** yee379 has quit IRC06:12
*** yee379 has joined #openstack-kolla06:12
*** yee379 has quit IRC06:14
*** yee379 has joined #openstack-kolla06:15
*** tovin07_ has joined #openstack-kolla06:18
*** sayantani01 has quit IRC06:25
*** sayantani01 has joined #openstack-kolla06:26
openstackgerritjimmygc proposed openstack/kolla-ansible master: Fix glance Upload large image haproxy timeout  https://review.openstack.org/44944006:30
*** MasterOfBugs has quit IRC06:32
*** yee379 has quit IRC06:44
*** yee379 has joined #openstack-kolla06:44
*** jrobinson has quit IRC06:47
*** targon has joined #openstack-kolla06:52
*** unicell has joined #openstack-kolla06:57
sdakezhubingbing around?06:57
*** unicell1 has quit IRC06:57
zhubingbinghear06:57
zhubingbing;)06:58
*** jascott1_ has quit IRC07:01
*** jascott1_ has joined #openstack-kolla07:01
spsuryaJeffrey4l: ping....07:05
*** jascott1_ has quit IRC07:06
sdakezhubingbing any progress on https://review.openstack.org/#/c/439740/07:07
spsuryasdake there many BPs which are still in ocata/newton series07:10
spsuryashould not we move these to pike ?07:10
sdakespsurya not sure - i dont work on kolla-ansible all that much anymore07:10
sdakespsurya i dont do triage for the blueprints/bugs in that deliverable07:11
spsuryasdake: there are many in kolla deliverable too07:11
sdakespsurya i dont do triage at all for bugs in kolla or kolla-ansible deliverable unless someone specifically asks me to do so07:12
spsuryasdake: roger07:12
sdakespsurya i barely do triage for blueprints07:12
sdakespsurya i have alot on my plate at present, and there are others in the various drivers team that can do that triage07:12
spsuryasdake: i was just getting aware of that :(07:13
sdakespsurya how do you mean07:13
spsuryasdake: i wanted know ? Is there any specific reason for the BPs which are in the ocata series07:14
spsuryasdake: my apology. If i asked something wrong :(07:16
sdakespsurya no reason to apologize07:17
sdakespsurya i dont know why people put blueprints in ocata - people self organize in kolla07:17
sdaketypically in the past ocata blueprints that didn't finish should be removed from ocata and retargeted towards pike07:18
duonghqhmm, bare kubeadm is way different to kargo07:18
spsuryasdake : roger, i was also thinking the same07:19
spsuryathanks07:19
sdakespsurya if your not in the drivers team i can ad dyou and you can do that retargeting07:20
sdakeanyone in the drivers team shoudl be doing that work07:20
sdakewe shouldn't have SPOFs in our community around any specific individual07:20
spsuryasdake:  i am in already in driver team07:21
spsuryai will move them to pike-107:21
sdakespsurya well then you can retarget blueprints :)07:21
*** manheim has joined #openstack-kolla07:22
*** manheim has quit IRC07:22
*** manheim has joined #openstack-kolla07:23
spsuryaroger ......))))))07:23
duonghqI have meeting now, BRB07:24
*** sayantani01 has quit IRC07:24
*** jascott1_ has joined #openstack-kolla07:25
*** jmccarthy has joined #openstack-kolla07:26
sdakebedtime07:29
sdakelater07:29
*** caoyuan_ has joined #openstack-kolla07:31
*** Margin is now known as MarginHu07:32
*** caoyuan has quit IRC07:33
spsuryaJeffrey4l: done for https://review.openstack.org/#/c/432300/1307:33
*** satyar has joined #openstack-kolla07:33
*** zhuzeyu has joined #openstack-kolla07:41
*** jascott1_ has quit IRC07:44
*** jascott1_ has joined #openstack-kolla07:45
*** jascott1_ has quit IRC07:50
openstackgerritMarcin Juszkiewicz proposed openstack/kolla master: ironic-pxe: handle non-x86 architectures  https://review.openstack.org/43481707:54
hrwmorning07:54
openstackgerritMarcin Juszkiewicz proposed openstack/kolla master: debian: enable all images enabled for Ubuntu  https://review.openstack.org/43278707:54
openstackgerritMarcin Juszkiewicz proposed openstack/kolla master: Add support for non-x86 architectures (aarch64, ppc64le)  https://review.openstack.org/43094007:54
openstackgerritMarcin Juszkiewicz proposed openstack/kolla master: openstack-base/kolla-toolbox: use mariadb-devel for CentOS and !x86-64  https://review.openstack.org/44483207:54
openstackgerritMarcin Juszkiewicz proposed openstack/kolla master: nova-libvirt: handle ppc64le  https://review.openstack.org/43481007:54
openstackgerritMarcin Juszkiewicz proposed openstack/kolla master: ceph: move ceph-fuse package to cephfs-fuse image  https://review.openstack.org/44063407:54
openstackgerritMarcin Juszkiewicz proposed openstack/kolla master: kubernetes: disable for architectures other than x86-64  https://review.openstack.org/44563807:54
openstackgerritMarcin Juszkiewicz proposed openstack/kolla master: nova-compute: handle rtslib(-fb) for debian-binary builds too  https://review.openstack.org/43594107:54
openstackgerritMarcin Juszkiewicz proposed openstack/kolla master: debian: move to stretch  https://review.openstack.org/43445307:54
openstackgerritpangliye proposed openstack/kolla-ansible master: Update configuration of Telegraf about mysql  https://review.openstack.org/44946507:54
*** matrohon has joined #openstack-kolla07:54
*** jaosorior has joined #openstack-kolla07:55
hrwcan someone review/+2 https://review.openstack.org/449135 and then https://review.openstack.org/449086 one? properly working yum is important07:55
*** yangyapeng has joined #openstack-kolla07:56
openstackgerritJuan Antonio Osorio Robles proposed openstack/kolla master: RHEL: Add mod_ssl for services running over httpd  https://review.openstack.org/44691108:02
*** shardy_afk is now known as shardy08:09
duonghqhrw, done for the 1st08:11
hrwduonghq: thx08:12
duonghqhrw, can you recheck the 2nd one08:12
*** mgoddard has joined #openstack-kolla08:12
hrwduonghq: it requires 1st one to be merged to pass08:12
duonghqah, yeah, need waiting for it08:13
hrwduonghq: I made 2nd because 1st failed08:14
duonghqhrw, roger08:14
duonghqhrw, need leaving now, cya08:18
hrwcu duonghq08:21
*** duonghq has quit IRC08:21
*** mnasiadka has joined #openstack-kolla08:23
*** caowei has joined #openstack-kolla08:30
*** manheim has quit IRC08:33
*** matrohon has quit IRC08:37
*** mgoddard has quit IRC08:40
*** matrohon has joined #openstack-kolla08:41
*** zhubingbing has quit IRC08:44
*** zhubingbing has joined #openstack-kolla08:45
*** magicboiz has quit IRC08:45
*** sean-k-mooney has quit IRC08:48
*** athomas has joined #openstack-kolla08:57
*** zhubingbing has quit IRC09:02
*** manheim has joined #openstack-kolla09:03
*** MarginHu has left #openstack-kolla09:06
*** zhubingbing_ has joined #openstack-kolla09:09
zhubingbing_sdake09:09
zhubingbing_i am not begin, i will start in this nighhe09:10
zhubingbing_night09:10
*** yuanying has quit IRC09:12
*** magicboiz has joined #openstack-kolla09:16
*** mgoddard has joined #openstack-kolla09:18
openstackgerritMerged openstack/kolla master: ironic-conductor: add missing comma for centos/source build  https://review.openstack.org/44913509:23
*** zhubingbing_ has quit IRC09:24
*** zhubingbing_ has joined #openstack-kolla09:25
*** pbandark has joined #openstack-kolla09:29
*** zhubingbing_ has quit IRC09:32
*** jrist has quit IRC09:34
openstackgerritjimmygc proposed openstack/kolla-ansible master: Add VMware vSphere support  https://review.openstack.org/44952909:35
*** zhubingbing_ has joined #openstack-kolla09:37
*** caowei has quit IRC09:37
*** egonzalez has joined #openstack-kolla09:40
openstackgerritEduardo Gonzalez proposed openstack/kolla stable/newton: ironic-conductor: add missing comma for centos/source build  https://review.openstack.org/44953509:41
*** zhuzeyu has quit IRC09:43
*** deadnull has joined #openstack-kolla09:43
*** bmace has quit IRC09:48
*** skramaja_ has joined #openstack-kolla09:51
*** rabel has joined #openstack-kolla09:52
openstackgerritjimmygc proposed openstack/kolla-ansible master: Add VMware vSphere support  https://review.openstack.org/44952909:54
*** skramaja has quit IRC09:54
*** yuanying has joined #openstack-kolla09:54
*** sean-k-mooney has joined #openstack-kolla09:54
openstackgerritjimmygc proposed openstack/kolla master: Add vSphere Support  https://review.openstack.org/44939509:55
*** Serlex has joined #openstack-kolla09:57
*** bmace has joined #openstack-kolla10:01
openstackgerritzhubingbing proposed openstack/kolla-kubernetes master: Fix problem with kollakube res create configmap  https://review.openstack.org/43974010:04
*** pcaruana has joined #openstack-kolla10:05
*** zhubingbing_ has quit IRC10:06
*** cuongnv has quit IRC10:07
*** dixiaoli has joined #openstack-kolla10:08
*** skramaja_ is now known as skramaja10:10
*** tovin07_ has quit IRC10:11
*** egonzalez has quit IRC10:15
*** pbourke has quit IRC10:17
*** pbourke has joined #openstack-kolla10:18
*** manheim has quit IRC10:24
*** manheim has joined #openstack-kolla10:26
*** matrohon has quit IRC10:30
*** pbandark has quit IRC10:31
*** manheim has quit IRC10:35
*** manheim has joined #openstack-kolla10:36
hawihi. i do have question - i deployed some machines and because quota limits it failed. now overview shows that memory is in use and cpus are in use, but actually i cant see failed instances.10:36
hawiso - how to free resources again?10:37
*** Pavo has joined #openstack-kolla10:38
hawii guess something was updated in mariadb and i need to fix it manually?10:39
*** zhurong has quit IRC10:40
*** matrohon has joined #openstack-kolla10:41
*** pbandark has joined #openstack-kolla10:44
*** athomas has quit IRC10:50
*** athomas has joined #openstack-kolla10:51
*** athomas has quit IRC10:52
*** zhubingbing_ has joined #openstack-kolla10:53
openstackgerritjimmygc proposed openstack/kolla master: Add vSphere Support  https://review.openstack.org/44939510:57
*** athomas has joined #openstack-kolla10:58
*** dixiaoli has quit IRC11:03
*** Pavo has quit IRC11:04
*** magicboiz has quit IRC11:07
*** rwallner has joined #openstack-kolla11:10
openstackgerritPaul Bourke (pbourke) proposed openstack/kolla master: Fix source install for vmtp  https://review.openstack.org/44851511:10
openstackgerritjimmygc proposed openstack/kolla master: Add vSphere Support  https://review.openstack.org/44939511:11
jaosoriorHey guys, if you have time could you check these out https://review.openstack.org/#/q/status:open+project:openstack/kolla+branch:master+topic:bug/1675308 ?11:11
*** tvignaud has quit IRC11:16
*** Pavo has joined #openstack-kolla11:18
*** magicboiz has joined #openstack-kolla11:18
*** tovin07 is now known as tovin07_at_work11:20
*** tovin07_at_work has quit IRC11:20
pbourkejaosorior: done11:23
jaosoriorthanks pbourke!11:23
*** Pavo has quit IRC11:25
spsuryasup guys11:27
*** tvignaud has joined #openstack-kolla11:29
*** tvignaud has quit IRC11:37
*** blallau has joined #openstack-kolla11:41
*** caoyuan_ has quit IRC11:49
*** tvignaud has joined #openstack-kolla11:51
*** Serlex has quit IRC11:54
*** tvignaud has quit IRC11:58
jmccarthyHmm ok ansible question, in the keystone bootstrap.yml task, there is a conditional include at the end for bootstrap_service.yml when:database.changed. database is registered in the first task 'Creating keystone database' - but I dont see any 'changed_when' or - how does it know it's changed ?11:59
openstackgerritjimmygc proposed openstack/kolla master: Add vSphere Support  https://review.openstack.org/44939512:08
spsuryaJeffrey4l: ping...12:10
Jeffrey4lspsurya, pong but sorry, busy.12:11
spsuryaJeffrey4l: np ;)12:11
*** tvignaud has joined #openstack-kolla12:12
spsuryawhenever you got few moment please reply to this12:12
spsuryaJeffrey4l:12:12
Jeffrey4lspsurya, reply which ?12:13
spsuryaI was re-targeting the all the kolla and kolla-ansible to pike12:13
*** rabel has left #openstack-kolla12:13
spsuryai.e all BPs12:14
Jeffrey4lspsurya, please...12:14
Jeffrey4li can add u to kolla-drivers. then you can do this.12:14
spsuryaJeffrey4l: please check12:14
spsuryaI think i am already there12:15
Jeffrey4lspsurya, yes. you are.12:16
Jeffrey4lthen you have permission to do this.12:16
*** fooliouno has quit IRC12:16
spsuryaJeffrey4l: yes12:17
spsuryaso moving all to pike-112:17
Jeffrey4lthansks ;)12:17
*** tvignaud has quit IRC12:18
*** manheim has quit IRC12:20
*** manheim has joined #openstack-kolla12:21
*** mliima has joined #openstack-kolla12:23
openstackgerritBertrand Lallau proposed openstack/kolla-ansible master: Remove Ubuntu 14.04 related code  https://review.openstack.org/44879212:23
*** schwicht has joined #openstack-kolla12:24
mliimamorning guyas12:27
*** srwilkers has joined #openstack-kolla12:29
*** schwicht has quit IRC12:30
*** yangyapeng has quit IRC12:31
*** tvignaud has joined #openstack-kolla12:31
*** shardy is now known as shardy_lunch12:31
*** caoyuan has joined #openstack-kolla12:32
*** schwicht has joined #openstack-kolla12:34
openstackgerritMerged openstack/kolla stable/mitaka: Updated from global requirements  https://review.openstack.org/44872312:35
*** jascott1_ has joined #openstack-kolla12:36
*** caoyuan_ has joined #openstack-kolla12:36
*** caoyuan has quit IRC12:36
srwilkersmorning12:36
portdirectmorning mliima o/12:36
spsuryamliima: morning12:38
*** jascott1_ has quit IRC12:41
*** schwicht has quit IRC12:47
*** Serlex has joined #openstack-kolla12:52
*** rwallner has quit IRC12:55
*** david-lyle has quit IRC12:55
kolla-slack<v1k0d3n> morning everyone12:55
portdirect'morning v1k0d3n. sup dude?12:56
*** lamt has joined #openstack-kolla12:56
*** goldyfruit has joined #openstack-kolla12:56
v1k0d3nnothing portdirect...just reading over things, catching up. TGIF.12:57
portdirectlol - just had to look up TGIF. Indeed :D12:57
*** rwallner has joined #openstack-kolla12:57
v1k0d3nhaha welcome to 'merica buddy! :-)12:58
*** magicboiz has quit IRC13:01
*** tvignaud has quit IRC13:03
sdakemorning peeps13:07
*** schwicht has joined #openstack-kolla13:08
sdakesbezverk_ ping13:09
sbezverk_sdake: pong13:14
sdakesbezverk_ there was a report of ironic not working properly (cash loop backoff) related to dependenciesnot setup in the services chart properly13:15
sdakering any bells?13:15
sdakesbezverk_ heard it third hand so ymmv13:15
*** mbruzek has joined #openstack-kolla13:15
spsuryasdake: morning13:16
*** tvignaud has joined #openstack-kolla13:16
sdakehey spsurya13:16
sbezverk_sdake: at the gate?13:16
sdakesbezverk_ i think it was not at the gate13:16
sdakesbezverk_ I looked at the pod ith describe13:17
sdakeand it didnt'have any depednencies defined at all13:17
*** shardy_lunch is now known as shardy13:17
sbezverk_sdake: then it must be config issue, my cluster and the gate shows green..13:17
sdakesbezverk_ if you can login to my boxes for 3 mins and look that might be h elpful13:17
sbezverk_sdake: sorry, I have few things I need to finish13:18
sbezverk_sdake: I can try in the afternoon the earliest13:19
sdakesbezverk_ ok thanks13:19
sdakesbezverk_ note this was with 4.0.0 images13:19
sdakewhich are not gated at all13:19
spsuryaJeffrey4l: sdake: re-targeted all the BPs of kolla and kolla-ansible to pike-1 which were approved and  Essential or High or medium priority and the one which was got assignee13:19
sdakeI bought a slew of 96khz/24bit msuci last night for 20% off13:21
*** yangyapeng has joined #openstack-kolla13:21
*** manheim has quit IRC13:21
spsuryasdake: nice13:21
*** jrist has joined #openstack-kolla13:22
sbezverk_sdake: gate jobs run 4.0.0 images13:23
sbezverk_I mean for ironic13:23
sdakespsurya yup i load it up on my phone for when i have to get on a plane and decode it with: https://www.amazon.com/OPPO-HA-2SE-Portable-Headphone-Amplifier/dp/B01LG8FWSY/ref=sr_1_4?s=audio-video-accessories&ie=UTF8&qid=1490361822&sr=1-413:24
sdakespsurya hooks up to my phone via OTG USB13:24
sdakesbezverk_ thats good to hear - so possibly a problem in my environment13:24
sdakesbezverk_ what is wierd is the same exact thing works with 3.0.3-beta1 images13:24
sdakesame exact everything - cloud.yaml etc13:25
openstackgerritMerged openstack/kolla-ansible master: Delete the deprecated option from the group "DEFAULT"  https://review.openstack.org/44736813:29
openstackgerritMerged openstack/kolla-ansible master: change log-* to flog-*.  https://review.openstack.org/44938413:29
sbezverk_sdake: there were tons of changes on ironic side introduced by latest PS, I am not surprised. Check my cloud yaml I posted yesterday it has all setting you need to run ironic13:30
*** jtriley has joined #openstack-kolla13:31
*** zhubingbing_ has quit IRC13:36
*** zhubingbing_ has joined #openstack-kolla13:39
hawii wonder if there is some trick to make central logging working on otaca? i enabled it in globals.yml, got containers on control nodes, but it seems that something wrong with elastic config (should i configure data dir there?) - at least kibana doesnt find log-*13:42
hawialso i tried to enable grafana - nothing happened at all. what i need to enable at same time, if i say "grafana=yes"?13:42
*** manheim has joined #openstack-kolla13:42
*** skramaja has quit IRC13:48
*** blallau has quit IRC14:01
*** mdnadeem has quit IRC14:06
mnasermorning14:07
*** dave-mccowan has joined #openstack-kolla14:07
*** lamt has quit IRC14:09
openstackgerritMohammed Naser proposed openstack/kolla master: Merged multiple ENV into a single layer  https://review.openstack.org/44937114:14
*** satyar_ has joined #openstack-kolla14:17
*** satyar has quit IRC14:18
*** jbadiapa has quit IRC14:18
*** shashank_t_ has joined #openstack-kolla14:22
*** david-lyle has joined #openstack-kolla14:26
jaosoriormnaser, sdake hey guys, if you have time could you check these out https://review.openstack.org/#/q/status:open+project:openstack/kolla+branch:master+topic:bug/1675308 ?14:30
*** jbadiapa has joined #openstack-kolla14:33
*** matrohon has quit IRC14:33
mnaserjaosorior i know you did a lot of work on this14:35
mnaserbut im a bit curious14:35
jaosoriormnaser: what's up?14:35
mnaserif we can remove a lot of deduplication by adding like14:35
mnaseropenstack-base-http14:35
mnaseri know you did a lot of work on those patches but maybe we can remove a lot of deduplication with that? (i'd hate to have to rework all that i know you put in a lot of effort into it)14:36
mnaserbut again im not a core so its just my "personal thought"14:36
jaosoriorMAYBE that could work? I honestly don't know the code well enough to figure out if the configurations all match up and if we could do that.14:37
mnaserjaosorior i can imagine of a few reason why it wouldnt right now14:37
jaosoriorpretty new to working with kolla myself14:37
mnaserex: nova-api and nova-placement-api need httpd14:37
mnasernova-*api base off of nova-base14:37
jaosorioris nova-api running over httpd?14:38
mnaserhttps://review.openstack.org/#/c/446911/8/docker/nova/nova-api/Dockerfile.j2 i guess?14:38
jaosoriorcause as far as I knew that was broken14:38
mnaseraccording to your patch :-P14:38
jaosorioror which nova component was it...14:38
*** jascott1_ has joined #openstack-kolla14:38
mnaserid hope its not14:39
jaosoriormnaser: well, you would have noticed. Since it was "supposed" to be broken for around a month now14:39
jaosoriormnaser: it might have been fixed, which means that I could start re-enabling running nova-api over httpd in TripleO14:40
mnaserjaosorior mm im not sure, in prod we've stuck to running nova-api for now14:41
jaosoriormnaser: so you don't run nova-api over containers?14:41
mnaserjaosorior we do, but stable/newton14:41
mnaserso i believe that uses nova-api14:41
jaosoriorah, that makes sense14:42
jaosoriorso yeah, nova-api over httpd in pike is currently not working properly AFAIK. Or at least with the configuration and requirements we have in TripleO14:42
*** jascott1_ has quit IRC14:43
*** targon has quit IRC14:47
*** erhudy has quit IRC14:50
jemcevoyinc0:  I have a new issue.   kolla-build is failing on the customer site because docker build does not have the http/https man-in-the-middle proxy configured inside the centos7  containers so the rpm --import https://<blah> fails because it needs access to the ca.crt file for the proxy.   Any ideas on how to get around this?14:51
*** yangyapeng has quit IRC14:51
*** matrohon has joined #openstack-kolla14:51
*** yangyapeng has joined #openstack-kolla14:52
*** haplo37 has quit IRC14:55
*** jbadiapa has quit IRC14:55
openstackgerritMerged openstack/kolla-ansible master: Designate: manage Api and Central services workers  https://review.openstack.org/44492914:56
*** jbadiapa has joined #openstack-kolla14:56
*** esharao has joined #openstack-kolla14:58
esharaoHello.. im trying to bring up kolla kubernetes... and i was succuessfully able to bring up all pods except nova-compute and nova-libvirt.. it keeps on crashing.. any advice as to how to proceed?14:59
*** lucasxu has joined #openstack-kolla14:59
mandrejaosorior, mnaser: I like the idea of an openstack-base-http image, that way we would also be able to save a lot of memory on the host if I'm not mistaken15:00
mandreone alternative (to reduce code duplication, but not bits duplication) would be to write an jinja2 macro that installs the required http related packages15:01
esharaowhen i look at the logs from k8s dashboard i see the last statement in the log as - Running command: 'nova-compute'15:03
sdakeesharao which version of the contianers are you using15:04
sdakeesharao i find the same result with 4.0.0 contianers15:04
sdakesbezverk_ has indicated this is because the cloud.yaml is incorrect15:05
esharao4.0.015:05
sdakeesharao how are you deploying?15:05
*** haplo37 has joined #openstack-kolla15:05
esharaoi used the kolla-kubernetes to deploy15:05
esharaoi also tried using HELM and it also failed for nova-compute15:06
*** fooliouno has joined #openstack-kolla15:09
*** mnasiadka has quit IRC15:10
sdakeesharao what I mean is what set of tools did you use to deploy15:11
sdakeesharao or doucmentation15:11
*** blallau has joined #openstack-kolla15:12
esharaoi followed the docs (Quickstart guide)15:13
esharaowith some changes to files (removal of petsets, etc)15:13
sdakeesharao i see - those docs are really old - may I recommend: https://review.openstack.org/#/c/447356/15:14
sdakeesharao i am going to submit a review now to move those docs to a deprecated section15:14
sdakeesharao note these docs are incomplete15:14
sdakeesharao the one in the review15:14
sdakeesharao testing wanted :)15:14
sdakeesharao you probbably spent weeks trying to comb through the old docs to get helm operational15:15
sdakeesharao atleast i did15:15
*** david-lyle has quit IRC15:15
esharaoyep...15:15
esharao3 weeks15:15
esharaoand was able to get it up based on multiple docs.. and stuff..15:15
esharaoi can delete my whole stuff and recreate following this review doc15:15
esharaoand will let you know15:15
esharaoi only have a single machine.. will this work?15:16
esharaoand can i use kube-deploy to get my kubernetes up or do i need to use kubeadm?15:17
rwellumsdake: that's a great qn from esharao  - why is 'deployment' hinged on 2-machines?15:17
mnaserwiw has abyieb ever kiijed thriygh thus>15:22
mnaserhttps://github.com/grammarly/rocker15:22
mnaseroh boy15:22
mnaseri was index+1 on my keyboard15:22
mnaserwow has anyone ever looked through this?15:22
sdake2 machines - one is master one is minion15:23
sdakeI really believe we can taint or untaint the master so we can use one node15:23
sdakeI'd like to sort out how to do that15:23
*** gkadam has joined #openstack-kolla15:24
esharaoOk.. and what about kube-deploy vs kubeadm?15:24
sdakekube-deploy is what tool?15:24
sdakeis that kargo?15:24
esharaohttps://github.com/kubernetes/kube-deploy15:24
sdakeesharao the only delta we have found with deployment tools is the dns pod has a different name15:25
esharaohttps://docs.openstack.org/developer/kolla-kubernetes/kubernetes-setup.html15:25
esharaook..and ubuntu 16.04 for base os works .. i hope.. so will use that and follow this doc and see where it takes me.. thanks sdake15:26
sdakeesharao there are some rough spots int he docs that need tuning15:27
sdakeesharao specifically aroudn how to create a cloud.yaml file15:27
sdakeIbelieve rwellum is stuck there15:27
sdakealso there is a new cloudl.yaml file that sbezverk_ recommends to start with to keep the 4.0.0 images working: http://paste.openstack.org/show/603915/15:28
*** eanylin has joined #openstack-kolla15:28
esharaook..15:29
openstackgerritBen Swartzlander proposed openstack/kolla-ansible master: Add missing pip install to quickstart guide  https://review.openstack.org/44968515:29
sdakeesharao ubuntu should work for a base os, however, there are small incompatibilities - if your doing an eval i think its fine if your doing a production deploy - i'd recommend against mixing15:29
esharaoOk.. that makes sense..15:29
*** jaosorior has quit IRC15:30
*** hrw has quit IRC15:31
esharaobbiaf15:31
*** athomas has quit IRC15:33
*** caoyuan_ has quit IRC15:35
eanylinHi sdake, I used the instructions and were able to bring up the same set of services as seen in your video other than that for nova-compute.  You mentioned that there might be a problem with cloud.yaml?15:35
sdakeeanylin so instructions worked for you?15:36
eanylinit did15:36
sdakeeanylin did you have any modifications?15:36
eanylinI tested on Ubuntu 16.0415:36
*** hrw has joined #openstack-kolla15:36
eanylinI made modifications to correct the indentation for cinder15:36
sdakeeanylin how long did it take you to run through the instructions?15:36
eanylinas pointed out in the etherpad15:36
eanylinit took many nights, unfortunately :)15:36
eanylinbut once I got the right config15:36
eanylinit went pretty fast15:37
sdakecool - right config - this is where we are blocked ;)15:37
eanylinmaybe 1 to 2 hrs to get to the stage where nova-compute is crashing15:37
sdake1 or 2 hours isn't super but its pretty good :)15:37
eanylinyeah15:37
sdake(i mean as a general community thing not in your specific case)15:37
eanylinfrom base CentOS15:37
eanylinwith just docker15:37
eanylinI used the latest Docker CE15:37
eanylinsince I dont have the older version and I been tearing down my environment multiple times15:38
sdakeeanylin regarding the failing nova-compute15:38
eanylinhad to skip initial flight test to get the kubeadm init working15:38
eanylinok15:38
sdakeI know the cloud.yaml works well for 3.0.3-beta.1 images15:39
eanylinok15:39
sdakeas in it boots nova properly15:39
eanylinso I should not use tag 4.0.0 in cloud.yaml15:39
eanylinbut should use 3.0.3-beta.1?15:39
sdakeeanylin 4.0.0 works in the gate15:39
sdakeeanylin i'd recommend using 4.0.0 images15:39
eanylinI know your initial config was for 3.0.215:39
eanylinok15:39
eanylinso 4.0.0 is still the right image15:39
sdakeeanylin what we need to do is sort out why the 4.0.0 images aren't coming up properly15:40
sdakewe know from the gate the 4.0.0 images are good15:40
eanylinI see15:40
eanylinso the images are ok15:40
sdakethe 4.0.0 images are immutable15:40
eanylinbut somehow the config has some problems15:40
sdakeright15:40
eanylinand hence nova-compute is not coming up properly15:40
eanylinI see15:40
eanylinmakes senes15:40
sdakeand by config - I believe that is cloud.yaml :)15:40
eanylinsense*15:40
eanylinyeah, I think so15:40
sdakethis is my current line of analysis15:40
eanylinthats the one that I fed to helm15:41
eanylinbased on your sample :)15:41
sdakeeanylin are you using 2 nodes or 115:41
eanylinI only have 1 laptop actually15:41
eanylinwhere I used the wifi card and eth015:41
eanylinas the 2 interface15:41
eanylinwasnt sure if this is good enough15:41
sdakethat should work i think - but not certain15:41
eanylinI have another more powerful laptop that can create 2 VMs15:42
eanylinok15:42
sdakeone vm should be fine - i'm curious how you got controllers/computes on one vm15:42
eanylinwill VM work or we definitely need bare metal ?15:42
sdakerwellum had asked above15:42
eanylintainted it15:42
eanylinthen it will allow running of all containers isnt it?15:42
eanylinI got libvirt-bin up15:42
eanylinas in that container15:42
sdakeeanylin thanks - tainted it - could you add that to this review: https://review.openstack.org/#/c/447356/15:43
eanylinsure15:43
sdakeeanylin with exact steps you followed15:43
sdaketo taint it15:43
eanylinits part of the kubeadm set up15:43
eanylinif you look at the webpage to set up kubeadm15:43
sdakei must have skipped over that part of the docs :)15:43
eanylinhttps://kubernetes.io/docs/getting-started-guides/kubeadm/#instructions15:43
eanylinStep 2 says that we can taint it if we only have 1 node for dev work15:44
sdake# MASTER_NODE="actual_master_node_name"15:44
sdake# TAINT_KEY=$(kubectl get no ${MASTER_NODE} --template="")15:44
sdake# kubectl taint nodes ${MASTER_NODE} ${TAINT_KEY}-15:44
eanylinI think I used the old command15:45
eanylinnot this one15:45
eanylinthey updated this instructions as well I think15:45
sdakeused old command where - in that review?15:45
sdakeeanylin kolla-kubernetes has moved most of the deploy guide in the etherpad to the review here: https://review.openstack.org/#/c/447356/ - it has undergone some change as well15:46
eanylinI am quite sure the old ones work15:47
eanylinI tried the new one and I think I had problems15:47
eanylin# kubectl taint nodes --all dedicated-15:47
eanylinnode "test-01" tainted15:47
eanylintaint key="dedicated" and effect="" not found.15:47
eanylintaint key="dedicated" and effect="" not found.15:47
eanylinso this one will work15:47
eanylinyes, its in the review, it can be found in this link, http://deploy-preview-1321.kubernetes-io-vnext-staging.netlify.com/docs/getting-started-guides/kubeadm/15:47
eanylinwhich is mentioned15:47
*** vhosakot has joined #openstack-kolla15:48
sdakeso just: https://review.openstack.org/#/c/447356/15:48
eanylinso I was able to get to the same kind of failure with weave net as well as flannel deploy using master.sh (kube-deploy)15:48
eanylinI mena with nova-compute15:48
eanylinmean*15:48
sdakerathe so just : # kubectl taint nodes --all dedicated-15:48
eanylinyeah, that should work15:48
eanylinfor CentOS 715:48
eanylinand latest docker CE15:48
eanylinand kubeadm15:48
eanylinon single node15:48
sdakeeanylin nice!15:49
vhosakotsending this message from my IRC client to test if this is seen in my slack client... testing 1 2 3 a b c $%^&*{}[]15:49
kolla-slack<vhosakot> sending this message from my slack client to test if this is seen in my IRC client... testing 1 2 3 a b c $%^&*{}[]15:49
fooliounosdake: enaylin: FWIW, I brought up 4.0.0 images using the old workflow, and nova-compute is a crashloop there as well.15:49
eanylinok15:49
eanylinyeha, same15:49
eanylinthe video was very helpful, I can see it was working at that time15:49
sdakeyup it does indeed work :)15:50
sdakeits a config problem15:50
eanylinthe console shows "nova-compute" and crash15:50
eanylinyeah15:50
eanylinbrb15:50
vhosakothmm... if I'm logged into both IRC client and slack client, messaged sent from slack client are seen with username kolla-slack in my IRC client15:50
sdakevhosakot yup thats how the integration works15:51
rwellumfyi weave never worked for me, had to use canal or calico as an SDN. That was an issue for me in the developer guide - that it recommends weave that seems to be unreliable for most.15:51
vhosakotsdake: so, to chat in the #opensatck-kolla, should we use slack or IRC?15:51
sdakevhosakot your choice15:51
vhosakotI prefer IRC :)15:52
vhosakoti use IRC to chat in other IRC rooms like neutron, nova, glance, etc15:52
sdakeok - well I've got 1.5 hrs, wouldl folks be interested in a webex session with screen sharing to sort out the config bits?15:52
fooliounoeanylin: would you mind pasting the cloud.yml that worked for you on Ubuntu16.04 (on pastebin). Did you use kubernetes 1.5.4 or 1.5.2?15:52
vhosakotalso, the slack emojis are not seen in IRC client :)15:52
rwellumsdake: +115:53
fooliounovhosakot: Did you resolve your dashboard issues?15:54
*** Pavo has joined #openstack-kolla15:55
fooliounorwellum: Weave failed for me when ovs containers deployed15:56
vhosakotfooliouno: no, thanks for your help.  so, I've found that kubeadm needs service account creation for both dashboard and "helm init" to work so the secrets/certs are setup correctly.  I need to fix my service account in kubeadm before proceeding. I am making slow progress as I'm working on many things now :)15:56
*** manheim has quit IRC15:56
fooliounoWorked like a charm until then15:56
*** manheim has joined #openstack-kolla15:56
vhosakotfooliouno: https://kubernetes.io/docs/user-guide/secrets/#service-accounts-automatically-create-and-attach-secrets-with-api-credentials is what I need to do after deploying kubeadm.15:57
vhosakotfooliouno: do you have dashboard working on kubeadm?15:57
fooliounovhosakot: cool. Yes. I have kubeadm and dashboard.15:57
fooliounoLooks like its scheduled on the kube master right now. So its up. Will try to recreate it on a minion and see if it works.15:58
vhosakotfooliouno: did dashboard work out of the box for you?  you did "kubectl create -f https://raw.githubusercontent.com/kubernetes/dashboard/v1.6.0/src/deploy/kubernetes-dashboard.yaml", and boom the dashboard was up?15:58
fooliounoOne other thing. If you are using Flannel and docker 1.13, you may be tripped up by IP forwarding rules.15:58
vhosakotfooliouno: I'm using canal which sbezverk_ and kfox1111 recommended.15:59
fooliounoYes .. it did (when scheduled on master)15:59
*** krtaylor has joined #openstack-kolla15:59
fooliounoCanal may have the same issue since it uses Flannel underneath15:59
sdakei've got 1.5 hrs to go through the workflow on single node and sort out that review doc16:00
sdakeif folks want to join in, http://cisco.webex.com/join/stdake16:00
fooliounosdake: I will listen in16:00
mnaseroracle linux mirrors having any problems?16:00
mnaseroraclelinux jobs are timing out16:01
vhosakotfooliouno: how did you pin the dashboard pod to the master?  edit the dashboard sepc YAML and add label+nodeSelector?16:01
*** manheim has quit IRC16:01
sdakemnaser oracle linux uses rdo mirrors i believe16:01
mnaseronly for the openstack packages16:01
mnaser$os packages still use their mirros16:01
sdakemnaser also for infrastructure pakages16:01
fooliounoNo .. it was scheduled that way by kubernetes. If it fails, delete and retry16:01
vhosakotfooliouno: ah, when I re-try, it ends on one of the masters... I thought no pod is scheduled on the master except the k8s control-plane pods (scheduler, controller-manager, etcd, api-server), right?16:03
sdakefooliouno note you can login with a web-browser only so no need to download a client16:03
*** pcaruana has quit IRC16:03
vhosakotfooliouno: one of the workers I meant (not one of the masters)16:03
fooliounodashboard has atolerance to schedule on master16:03
vhosakotfooliouno: I read about it... where is the tolerence? let me read the dashboard spec..16:04
fooliounoIn the yaml16:04
sdakesrujan can you click "conenct to audio"16:05
sdakeanyone else interestedi n participating before i get started giving the instructions a dry run?16:05
sdakerwellum ?16:05
*** Pavo has quit IRC16:06
vhosakotfooliouno: is this the tolerence annotation --> https://github.com/kubernetes/dashboard/blob/master/src/deploy/kubernetes-dashboard.yaml#L38-L4616:06
vhosakotfooliouno: did you have to change this in the dashboard YAML?  -->  https://github.com/kubernetes/dashboard/blob/master/src/deploy/kubernetes-dashboard.yaml#L5916:07
eanylinhttp://pastebin.com/GUw5G2XJ16:08
vhosakotsdake: we're moving our lab now, I'll move me kubeadm cluster, fix my secrets, and ping when I get "helm init" working to continue16:08
eanylinits similar to what sdake did16:08
eanylinI used similar config16:08
vhosakotturns out "helm init" and dashboard both not working out-of-the-box with kubeadm is a pretty common problem I found.16:09
eanylinmy k8s server version is 1.5.516:10
*** Pavo has joined #openstack-kolla16:12
*** manheim has joined #openstack-kolla16:13
*** vhosakot_ has joined #openstack-kolla16:14
*** vhosakot has quit IRC16:14
fooliounovhosakot: sorry on sdake webex. will look and respond later16:16
vhosakot_cool. np16:18
*** skramaja has joined #openstack-kolla16:19
*** pbandark has quit IRC16:20
*** Pavo has quit IRC16:21
*** manheim has quit IRC16:22
*** manheim has joined #openstack-kolla16:22
*** lamt has joined #openstack-kolla16:23
*** mattmceuen has joined #openstack-kolla16:24
*** blallau has quit IRC16:24
*** manheim has quit IRC16:27
eanylin@vhosakot, you should be able to install dashboard by16:28
eanylinkubectl create -f https://rawgit.com/kubernetes/dashboard/master/src/deploy/kubernetes-dashboard.yaml16:28
eanylinbased on instructions from https://github.com/kubernetes/dashboard16:28
*** kolla-slack has quit IRC16:32
*** david-lyle has joined #openstack-kolla16:33
*** shashank_t_ has quit IRC16:35
*** shashank_t_ has joined #openstack-kolla16:35
*** jascott1_ has joined #openstack-kolla16:36
*** jascott1_ has quit IRC16:36
*** jascott1_ has joined #openstack-kolla16:42
*** MasterOfBugs has joined #openstack-kolla16:46
*** matrohon has quit IRC16:52
*** Serlex has quit IRC16:56
*** rwsu has quit IRC16:57
*** rwsu has joined #openstack-kolla17:00
*** Serlex has joined #openstack-kolla17:03
*** lamt has quit IRC17:17
vhosakot_eanylin: I did that, and it does not work with kubeadm :)17:18
*** unicell has quit IRC17:19
*** jrich523 has joined #openstack-kolla17:21
*** jmccarthy has left #openstack-kolla17:22
*** jrich523 has quit IRC17:22
eanylinI think I tested yesterday night, it was ok with version 1.6.017:23
eanylinimage: gcr.io/google_containers/kubernetes-dashboard-amd64:v1.6.017:23
eanylinyou can try and update the yaml file?17:23
*** schwicht has quit IRC17:26
*** Serlex has quit IRC17:27
*** sayantani01 has joined #openstack-kolla17:28
*** schwicht has joined #openstack-kolla17:28
*** jrich523 has joined #openstack-kolla17:28
*** lucasxu has quit IRC17:36
openstackgerritMichal Jastrzebski (inc0) proposed openstack/kolla-kubernetes master: Move glance config to kolla-k8s  https://review.openstack.org/44975517:38
*** unicell has joined #openstack-kolla17:45
*** manheim has joined #openstack-kolla17:49
*** manheim has quit IRC17:49
*** unicell has quit IRC17:49
*** unicell has joined #openstack-kolla17:50
*** manheim has joined #openstack-kolla17:50
*** mgoddard has quit IRC17:58
*** sayantani01 has quit IRC17:58
*** sayantani01 has joined #openstack-kolla17:59
*** zhubingbing_ has quit IRC18:00
*** skramaja has quit IRC18:00
*** Serlex has joined #openstack-kolla18:06
*** manheim has quit IRC18:14
*** manheim has joined #openstack-kolla18:15
*** gkadam has quit IRC18:15
*** manheim has quit IRC18:24
*** MasterOfBugs has quit IRC18:28
*** mgoddard has joined #openstack-kolla18:35
*** lamt has joined #openstack-kolla18:39
*** shashank_t_ has quit IRC18:40
*** MasterOfBugs has joined #openstack-kolla18:46
sbezverk_inc0: ping18:50
*** lucasxu has joined #openstack-kolla18:50
inc0whats up Serguey?18:50
*** shashank_t_ has joined #openstack-kolla18:50
sbezverk_inc0: do you know when this is going to be fixed? https://github.com/openstack/kolla-ansible/blob/master/ansible/roles/nova/templates/qemu.conf.j2#L318:51
sbezverk_inc0: it breaks ubuntu ironic deployment18:51
sbezverk_inc0: as I am getting permission denied when I try to access from the host libvirtd socket18:52
sbezverk_inc0: I see it in 4.0.0 images18:52
inc0hmm hard to tell18:58
sbezverk_inc0: :( cause you see, libvirt runs as nova in the container, but on the host nova user is not known19:00
sbezverk_inc0: maybe I could define user nova with the same user id and try to run virsh not with sudo bu with user nova id?19:01
jemcevoyinc0:  I have a new issue.   kolla-build is failing on the customer site because docker build does not have the http/https man-in-the-middle proxy configured inside the centos7  containers so the rpm --import https://<blah> fails because it needs access to the ca.crt file for the proxy.   Any ideas on how to get around this?19:02
inc0jemcevoy: well, you can use overrides19:04
inc0to inject files/configs to container buildtime19:04
inc0hold on19:04
inc0sbezverk_: I'll get back to you in a second, having few discussions at once and yours needs full attention;)19:05
inc0https://docs.openstack.org/developer/kolla/image-building.html#generic-customisation19:05
inc0jemcevoy: ^19:05
inc0jemcevoy: https://github.com/openstack/kolla/blob/master/docker/base/Dockerfile.j2#L16 you can extend this block to inject any piece of dockerfile-compatible code at the very beggining of base image19:07
inc0or look for block better suited for your needs19:07
Serlexsdake19:08
jemcevoyThat sounds like a perfect solution...  Looking into it.19:08
Serlexdid the second part of that webex happen?19:08
sdakeSerlex give me 20 mins19:08
sdakeSerlex and i'll get it going - need to feed myself19:08
sdakeserlex possibly 3019:08
*** tonanhngo has joined #openstack-kolla19:09
*** tonanhngo has quit IRC19:09
sbezverk_inc0: sure sure no rush19:10
Serlexok - but we should ask around if the others are interested as I was purely spectating19:12
inc0sbezverk_: right19:12
inc0so issue was vm files19:13
inc0with wrong permissions19:13
inc0my question is...why do you try to access libvirt from host?19:14
inc0I mean...we run everything in container19:14
*** mgoddard has quit IRC19:16
sbezverk_inc0: it is only at the gate when I need to define baremetal VM using virsh19:16
inc0well yeah adding user with same uid as used by nova in container might work19:17
sbezverk_inc0: since it is a very specific scenario I think the trick with the user should be good enough19:17
sbezverk_inc0: do you remember from the top of your head where or how user ids are getting generated, I think at one point you were going to use a static ids, is that right?19:18
inc0sbezverk_: https://github.com/openstack/kolla/blob/master/docker/base/Dockerfile.j2#L719:19
*** tonanhngo has joined #openstack-kolla19:19
inc0https://github.com/openstack/kolla/blob/master/kolla/common/config.py#L54319:19
sbezverk_inc0: right but where user.uid  comes from?19:19
inc0second link19:19
sbezverk_inc0: cool, missed it19:20
sbezverk_thanks a lot!19:20
kfox1111sbezverk_: can you exec into the container and use its virsh?19:20
*** erhudy has joined #openstack-kolla19:20
inc0alias virsh = docker exec -it nova_libvirt virsh :D19:21
inc0kfox1111: https://review.openstack.org/#/c/449254/19:21
kfox1111inc0: won't quite work.... isn't named that way. ;)19:22
kfox1111reviewing.19:22
*** yangyapeng has quit IRC19:24
sbezverk_kfox1111: hm, but then I need somehow dump xml into the container19:24
*** yangyapeng has joined #openstack-kolla19:24
sbezverk_kfox1111: adding nova user on the host with the same uid/guid should do the trick I think19:25
kfox1111sbezverk_: cat foo.xml | docker exec ... 'virsh <(cat)' ?19:26
inc0my god19:26
kfox1111hehe19:26
kfox1111shellfu. :)19:26
*** shashank_t_ has quit IRC19:27
*** shashank_t_ has joined #openstack-kolla19:27
inc0shell fu - bashing faces19:27
kfox1111:)19:28
inc0bash and boot19:28
sdakeok19:28
sbezverk_kfox1111: why not :)19:28
sdakenova-compute and nova-compute-ironic don't come up in my env19:28
sdakeany pointers?19:28
sdakewith 4.0.0 images19:28
sdakewith sbezverk_ 's cloud.yaml19:28
sbezverk_sdake: what they are complaning about?19:28
sdakethey enter running state then enter a crashlopbackoff19:28
sdakesbezverk_ not sure what to check19:29
sbezverk_sdake: what is the crash reason?19:29
sdakesbezverk_ run docker logs on em ?19:29
Serlexdid you create initramfs .kernel files sdake?19:29
sdakeSerlex not sure19:29
sbezverk_Serlex: initramfs is not needed for containers to run19:30
sbezverk_only to deply baremetal19:30
Serlexactually if it deployed, yeah you are past that requirement, my bad19:30
sdakesbezverk_ how should I debug?19:30
sdakesbezverk_ everyone in the community stuck at same point with 4.0.0 images19:31
sdakesbezverk_ that has managed to get past the deploy kubernetes gauntlet19:31
*** shashank_t_ has quit IRC19:31
sbezverk_sdake: why do you deploy ironic? have you tried just nova compute for fun and see if it is working?19:32
sbezverk_have you deployed nva placement19:32
sdakenova placement...19:32
sbezverk_have you deplyed nova cell0?19:32
sdakeletm e look19:32
kfox1111sdake: 4.0.0 images are.... lightly tested.19:32
sdakekfox1111 i understand19:32
kfox1111we need some gates for them.19:32
openstackgerritRich Wellum proposed openstack/kolla-kubernetes master: setup_gate.sh echos non-quoted params  https://review.openstack.org/44979019:32
sbezverk_sdake: the easiet is to take the latest deploy scripts from the gate and see what you missed19:33
sbezverk_I can guarantee you, you missed something19:33
sdakethe deploy scripts deploy 4.0.0?19:33
sbezverk_sdake: please check ironic gate job19:34
sdakesbezverk_ thanks19:34
sbezverk_you will see two main scripts one generic deployment with iscsi backend and second is ironic specific script19:34
sbezverk_just follwo them19:34
openstackgerritMichal Jastrzebski (inc0) proposed openstack/kolla-kubernetes master: Move glance config to kolla-k8s  https://review.openstack.org/44975519:35
sdakesbezverk_ is the placement api and cell0 in the nova service chart?19:36
sbezverk_nope19:36
sbezverk_because they will fail on old images19:36
sbezverk_I could not put them there19:36
sbezverk_you need to deploy them manually after the service19:37
jemcevoyinc0:  I had to fix a bug in base_yum_repo_keys in that same file:  The GPG key should be https://www.percona.com/downloads/RPM-GPG-KEY-percona not a local file... https://github.com/openstack/kolla/blob/master/docker/base/Dockerfile.j2#L6319:37
sbezverk_sdake: we can only put them in services after all images are 4.0.0 and higher19:37
sdakehow about we make a nova service chart for ocata19:38
sdakeor alternately remove 2.0 and 3.0 images :)19:38
kfox1111sbezverk_: we should put in a conditional dep.19:43
sbezverk_what will you check? in this dependency?? ;)19:48
inc0can we please remove 2 and 3 images? pretty pretty please?19:48
sbezverk_kfox1111: needs to confirm that with 4.0.0 no issues with ceph19:49
inc0why would it?19:49
sbezverk_inc0: it seems kfox1111 has laready tried in the past and there was a problem with cinder volumes19:52
sbezverk_kfox1111: unfortunately crazy bash thing will not work, as vbmc uses normal way to start/stop vm :(19:53
sdakekfox1111 how do you do a conditional dep20:06
inc0kfox1111: https://review.openstack.org/#/c/449755/ these gates seems to fail for reasons other than patchset..?20:08
openstackgerritSerguei Bezverkhi proposed openstack/kolla-kubernetes master: Fixing ironic deployment failure on Ubuntu  https://review.openstack.org/44980320:09
sdakekfox1111 need m0ar info on conditonal dep plz20:10
sbezverk_sdake: making it conditional will not help much, because there is no way reliably identify what images you use.20:12
sbezverk_they can be overridden at multiple levels.20:13
sdakewe could set a "deploy_ocata" field20:13
sbezverk_sdake: and in pike we change it for deploy_pike??? it does not sound as a good solution20:13
sdakea solution that works isbetter then no solution at all :)20:14
sbezverk_why do not you deploy it just after that manually ?? it is not big deal..20:14
sbezverk_until we sort out images issue20:14
sdakewe will always have this problem of images being added to services in new versions20:14
sdakewhy have a nova service chart if some of the services must be deployed manually?20:15
sdakedeploying manually a bunch of extra services is not very intuitive (to answer your question)20:16
sdakeand compute-kit won't deploy the whole schebang20:16
sdakehelm install compute-kit20:16
sdakebam - 4.0.0 images should work with that use casse20:16
sbezverk_sdake: we have not release kolla-kube20:16
sbezverk_so now it is used mostly by developers20:17
sbezverk_of course at the release time all these will be sorred out20:17
sdakehow about a legacy_images flag20:17
sdakethat points backwards instead of deploy_ocata which is a point in time20:18
sbezverk_man :( 1.5.5 kube is broken all gates are red20:18
sbezverk_we should stop pulling for the latest kube20:19
sbezverk_kfox1111: what do you think?20:19
sdakethat kind of thiking leads up down the road of keeping 2.0.0 images in the tree ;)20:19
sbezverk_sdake: yeah of course it is much better to have all gate jobs red20:20
sdakei didn't say it was20:20
sdakea temporary pin is fine - a permanent pin not fine - temporary often turns into permanent in kolla-kubernetes...20:20
*** mliima has quit IRC20:21
*** fooliouno has quit IRC20:31
sdakeinc0 when you get bored, you might have a look at updating this: https://review.openstack.org/#/c/346455/20:32
*** rwallner has quit IRC20:33
sdakeinc0 ttx looking for an update and your it :)20:35
rwellumsdake: I missed k8's meeting part 2 and it all works right? :p20:35
sdakerwellum i haven' theld a part 220:35
sdakei was thinking on the hour we can do that - had some things creep into my schedule20:36
rwellumsdake: It's Friday too...20:36
sdakerwellum I am pretty sure we have the info now to get past the problem everyone is suffering with wrt nova and 4.0.0 images20:36
sdakerwellum right - thats 5pm your time?20:36
sdakeor 4pm?20:36
rwellumDepends are we rounding up or down?20:37
rwellum4:3720:37
sdakeits 1;37 PM now where I am20:37
inc0sdake: I'll reup it today, thanks20:37
sdakeok so on the hour would be  5pm your time20:37
sdakethans inc020:37
inc0somehow I missed this review20:37
sdakerwellum is that too late to go - for 1 hour?20:37
sdakeinc0 yup happens to me all the time20:38
rwellumsdake: unless someone @home complains it's good.20:38
*** imcsk8 has quit IRC20:38
rwellumsdake: I thought you were MST for some reason. But you're further West I guess.20:39
sdakerwellum I am MST year round20:40
sdakealthough right now most people in M?T are MDT20:40
sdakeMDT is often abbreviated MT20:41
sdakeits confusing for everyone that is on daylight savings20:41
kfox1111what a day. :/20:41
sdakefor people not on daylight savings, its not confusing :)20:41
kfox1111all the computers are rebelling. :/20:41
sdakeatleast its not the 13th20:41
rwellumYeah I'd expect a 2 hr difference not 3 hr.20:41
kfox1111sdake: pretty sure it is somehow the 13th today. :/20:42
sdakekfox1111 and full moon :)20:42
kfox1111yeah. that too.20:42
kfox1111there are warewolfs out for sure. :/20:42
sdakerwellum if you type "what time is it in arizona" it pops out the proper time :)20:42
sdaketype into google that is20:42
kfox1111sdake: https://github.com/kubernetes/helm/issues/158520:43
rwellumOh right - that's not a cool feature of irc then...20:44
*** srwilkers has quit IRC20:44
sdakerwellum sorry for the duh moment :)20:45
*** imcsk8 has joined #openstack-kolla20:45
sdakerwellum timeanddate says MST: https://www.timeanddate.com/worldclock/usa/phoenix20:46
rwellumthat's actually pretty interesting... Phoenix has chosen not to observe DST.20:48
rwellumIt's a per-city choice?20:48
openstackgerritKevin Fox proposed openstack/kolla-kubernetes master: Helm Version File  https://review.openstack.org/44981420:51
*** matrohon has joined #openstack-kolla20:52
*** imcsk8 has quit IRC20:54
*** imcsk8 has joined #openstack-kolla20:56
*** jtriley has quit IRC21:00
*** dave-mccowan has quit IRC21:01
sdakerwellum all of az except parts of the reservation don't observe dst21:02
sdakesetting up webex now21:02
sdakefor folks that want to make a go at bringing up kubernetes + kolla_kubernetes in an AIO environment21:02
sdakecontinuation from this morning session21:02
sdakehttp://cisco.webex.com/join/stdake21:03
*** eaguilar has joined #openstack-kolla21:04
*** eaguilar has quit IRC21:05
sdakerwellum ^^21:06
*** pramodrj07 has joined #openstack-kolla21:16
*** mattmceuen has quit IRC21:18
esharaohttps://github.com/kubernetes/kubernetes/issues/3811821:19
*** MasterOfBugs has quit IRC21:20
*** jascott1 has quit IRC21:21
*** shasha_t_ has quit IRC21:21
*** pbandark has joined #openstack-kolla21:24
*** pbandark has quit IRC21:24
*** eaguilar has joined #openstack-kolla21:26
*** satyar_ has quit IRC21:42
kfox1111the evermind seems mad at me today.21:42
kfox1111all sorts of computers are rebelling.21:42
openstackgerritKevin Fox proposed openstack/kolla-kubernetes master: WIP: Helm Native Values.  https://review.openstack.org/44982621:44
dimskfox1111 : LOL21:45
openstackgerritKevin Fox proposed openstack/kolla-kubernetes master: WIP: Helm Native Values.  https://review.openstack.org/44982621:46
*** Pavo has joined #openstack-kolla21:48
eanylinhttps://review.openstack.org/#/c/439409/1/kolla_kubernetes/pathfinder.py21:49
openstackgerritKevin Fox proposed openstack/kolla-kubernetes master: Helm Version File  https://review.openstack.org/44981421:50
eanylinhttp://paste.openstack.org/raw/600976/21:55
*** Serlex has quit IRC21:59
*** sayantani01 has quit IRC22:01
*** pramodrj07 has quit IRC22:03
*** pramodrj07 has joined #openstack-kolla22:03
eanylin$ curl https://raw.githubusercontent.com/kubernetes/helm/master/scripts/get > get_helm.sh22:06
eanylin$ chmod 700 get_helm.sh22:06
eanylin$ ./get_helm.sh22:06
*** schwicht has quit IRC22:07
eanylinhttps://github.com/kubernetes/helm/blob/master/docs/install.md22:07
*** matrohon has quit IRC22:19
sbezverk_rwellum: ping22:21
*** esharao has quit IRC22:21
*** lucasxu has quit IRC22:27
*** tonanhngo has quit IRC22:30
*** schwicht has joined #openstack-kolla22:30
sbezverk_kfox1111: should we pin 1.5.4 at the gate until 1.5.5 gets fixed?22:34
kfox1111sure.22:38
kfox1111do you have a log of a broken 1.5.5 run?22:38
kfox1111nm. I've got a log.22:38
sbezverk_kfox1111: all jobs are failing22:40
sbezverk_I checked and I could not see any obvious reason other than api and other containers are restarting22:40
kfox1111weird. the changelog says its just a security fix.22:41
sbezverk_kfox1111: maybe bad build?22:41
kfox1111did canal change?22:42
sbezverk_do not think so, and it seems it was not restarting (surprisingly)22:42
kfox1111wondering if they changed ip's again.22:43
kfox1111nope. they are still the same...22:45
kfox1111so, kube-dns is consistently failing.22:50
sdakekube-dns renamed possibly?22:50
sdakenote i saw kube-dns consistently failing on canal earlier today22:51
sdakeit was not renamed however it was failing consistently22:51
sdakeand I can reproduce it locally22:51
kfox1111here we go22:51
kfox1111http://logs.openstack.org/14/449814/2/check/gate-kolla-kubernetes-deploy-centos-binary-2-iscsi-nv/f39afd8/logs/pods/kube-system-kube-dns-2924299975-91mp3-kube-dns.txt22:51
sdakeit is possible the gate problem is not kubernetes 1.5.5 however canal22:52
kfox1111its saying 1.6.0-alpha.....22:52
kfox1111possibly bad kubeadm22:52
sbezverk_kfox1111: you see kubedns might flap like a toilet seat, but it should not impact api-server22:52
kfox1111leaking in parts of 1.6.22:52
sbezverk_kfox1111: we have seen it many times before, but api-server was always rock solid22:53
sbezverk_at least in my experience22:53
kfox1111sbezverk_: where are you seeing api-server failing?22:54
kfox1111http://logs.openstack.org/14/449814/2/check/gate-kolla-kubernetes-deploy-centos-binary-2-iscsi-nv/f39afd8/console.html says it failed to start kube-dns.22:54
sbezverk_kfox1111: number of restarts for api-server is not 922:54
*** goldyfruit has quit IRC22:54
sbezverk_not 0 sorry22:54
kfox1111its running though.22:54
sdakei deployed this is what i am running: [sdake@kolla ~]$ rpm -qa | grep kubeadm22:54
sdakekubeadm-1.6.0-0.alpha.0.2074.a092d8e0f95f52.x86_6422:54
kfox1111and not restarting frequently.22:54
sbezverk_kfox1111: http://logs.openstack.org/90/449790/1/check/gate-kolla-kubernetes-deploy-centos-binary-2-ceph-nv/2d106b3/console.html22:55
sdakeand found that everything works fine with that versoin of kubeadm22:55
sdake(with weave)22:55
*** mbruzek has quit IRC22:55
*** eaguilar has quit IRC22:55
kfox1111sbezverk_: 1 is not a problem.22:55
sdakecalico, kube-dns fails22:55
kfox1111its responding enough to respond to the get pods request.22:55
kfox1111gate's failing because kubedns isn't fully up.22:55
kfox1111and I'm guessing thats because its a 1.6 kube-dns against a 1.5 cluster?22:56
kfox1111sdake: not sure about weave. never really deployed it.22:57
sbezverk_kfox1111: would be very surprised22:57
kfox1111sbezverk_: suprprised about what?22:57
sdakekfox1111 the point is kube-dns didnt start with calico, and kube-dns did start with weave22:57
sdakekube-dns + calico = crashloopbackoff; kube-dns + weave = golden22:58
kfox1111sdake: seen the exact opposite more often. :/22:58
sbezverk_Mar 24 19:47:48 localhost journal: E0324 19:47:48.646703       1 reflector.go:199] pkg/dns/dns.go:148: Failed to list *api.Service: Get https://172.16.128.1:443/api/v1/services?resourceVersion=0: dial tcp 172.16.128.1:443: i/o timeout22:59
kfox1111kubedns uses cni though, so if its not working, it can be network related.22:59
kfox1111sbezverk_: whats that from?22:59
kfox1111sbezverk_: and does it reconnect afterwards?22:59
sbezverk_messages22:59
kfox1111172.16... is that an address we gave it?23:00
*** erhudy has quit IRC23:00
kfox1111ah. yeah, it is.23:00
kfox1111but...23:01
kfox1111we do some sed'ing in setup_kubernetes.sh too for addresses.23:01
kfox1111maybe the sed is failing...23:01
kfox1111in tools/setup_kubernetes.sh23:01
*** schwicht has quit IRC23:01
sbezverk_kfox1111: it looks like cni is failing, damn23:02
sdakei suspect its the cni+calico connection23:03
sbezverk_kfox1111: scratch that at the very end of log it got net interface initialized23:04
*** jascott1 has joined #openstack-kolla23:04
kfox1111I0324 19:46:54.514390       1 manager.go:149] Using interface with name eth0 and address 104.239.174.20823:04
kfox1111that seems wrong....23:04
kfox1111well, maybe thats the nodes address...23:05
sbezverk_http://paste.openstack.org/show/604112/23:06
sbezverk_it might be ip address of physical box?23:07
*** eaguilar has joined #openstack-kolla23:07
sbezverk_kfox1111: yep it is a host at rackspace23:08
sbezverk_we should try to use cisco's contiv ;)23:09
kfox1111any cni plugin should work in theory.23:11
*** fooliouno has joined #openstack-kolla23:12
sbezverk_sure, the keyword is in theory ;)23:12
kfox1111so....23:12
kfox11111.5.5 was released the 21st.23:13
kfox1111so unlkely its was that.23:13
kfox11111.6.0rc1 was released today.23:13
kfox1111stuff broke today?23:13
sbezverk_kfox1111: are you sure yesterday's jobs were using 1.5.5??23:13
kfox1111sbezverk_: unsure. but would guess so.23:14
sbezverk_hm you are right, I check a job from 22nd23:15
sdakeany idea on this:   1h10s264{kubelet kolla}WarningFailedSyncError syncing pod, skipping: failed to "StartContainer" for "tiller" with ImagePullBackOff: "Back-off pulling image \"gcr.io/kubernetes-helm/tiller:v2.2.3\""23:15
sdake[sdake@kolla ~]$ docker pull gcr.io/kubernetes-helm/tiller:v2.2.323:15
sbezverk_and it was using 1.5.523:15
sbezverk_sdake: is there version 2.2.3????23:15
sdakeindeed23:16
sbezverk_I thought 2.2.2 is the latest23:16
sdakealso 2.2.2 fials to pull23:16
kfox1111sdake: do a docker pull manually on it.23:16
kfox1111sometimes tells you more about it.23:16
sdakei did kfox1111 : [sdake@kolla ~]$ docker pull gcr.io/kubernetes-helm/tiller:v2.2.323:16
sdakeError response from daemon: target is unknown23:16
kfox1111weird.23:17
kfox1111what docker version?23:17
sdake1.3.023:17
sdake1.13.023:17
sbezverk_gcr.io/kubernetes-helm/tiller:v2.2.223:17
sbezverk_works for me23:17
kfox11111.13.0 is broken I think.23:17
kfox1111[kfox@mantis ~]$ docker pull gcr.io/kubernetes-helm/tiller:v2.2.323:17
kfox1111Trying to pull repository gcr.io/kubernetes-helm/tiller ...23:17
kfox1111v2.2.3: Pulling from gcr.io/kubernetes-helm/tiller23:17
kfox111153ebc9bfbcc0: Pull complete23:17
sdake[sdake@kolla ~]$ docker pull gcr.io/kubernetes-helm/tiller:v2.2.223:17
sdakeError response from daemon: target is unknown23:17
kfox1111I think 1.13.0 broke the docker hub protocol.23:17
* sdake groans23:18
kfox1111see if there is a 1.13.123:18
kfox1111or revert to the distro provided docker.23:18
sbezverk_could be I run 1.12.6 docker23:18
kfox1111I'm on 1.10.3-44.el7.centos23:18
kfox1111docker inc made changes in 1.13 that broke google's repo.23:19
sbezverk_kfox1111: no wonder you like 2.0.2 images ;)23:19
*** vhosakot_ has quit IRC23:19
kfox1111sbezverk_: no, in this case, I prefer redhat's lack of breaking stuff, over dockerinc's habbit of breaking thigns. ;)23:19
kfox1111sdake's seeing why first hand. :/23:20
sbezverk_kfox1111: I get docker from epel I think and they keep it at 1.12.623:20
kfox1111sbezverk_: ah. yeah, they may have a newer version now. this test box hasn't ben updated in a little while.23:20
kfox1111oh. yeah, in fact, I was mid update on this box when it ate itself..... thats right...23:21
kfox1111I probaby shouldn't restart the update today... the evermind's not being kind to me today...23:22
sbezverk_kfox1111: yeah, good idea, no risky operations on friday afternoon ;)23:22
kfox1111too late. docker = 1.12.6 :)23:23
kfox1111and it pulls tiller:v2.2.3 ok.23:23
kfox1111sdake: see if you can upgrade past 1.13.0.23:23
kfox1111that one I think is a month+ old. maybe they have a fix by now.23:24
sdakethis seems to work:  Version:      17.03.0-ce23:26
kfox1111ah. yeah. they seem to be going more freemium. :/23:27
*** tonanhngo has joined #openstack-kolla23:30
kfox1111sbezverk_: I'm seeing kubelet 1.5.4 from the repo when I pull it local.23:32
kfox1111kubeadm is v1.6.0-alpha.0.2074+a092d8e0f95f52, but built on 2016-12-13T17:03:18Z23:32
kfox1111so not sure anything changed there.23:32
sbezverk_kfox1111: a couple of day ago new version of cni was released23:34
*** tonanhngo has quit IRC23:34
sbezverk_maybe we could pull it into the gate job and test23:35
sbezverk_if the issue is the same, then it is somethign else23:35
sdakesbezverk_ how does this look for a cloud.yaml: http://paste.openstack.org/show/604113/23:36
kfox1111just wondering what actuallyl changed...23:36
sdakekfox1111 ^^23:36
kfox1111sdake: looks ok.23:36
kfox1111well, unquote the port_externals I think.23:37
sbezverk_kfox1111: cni we install is very very old23:38
sbezverk_0.3.1 but the current stable is 0.5.123:38
kfox1111sbezverk_: there's another possibility...23:39
kfox1111ironic merged today?23:39
sbezverk_kfox1111: nope23:39
sbezverk_last night, after that inc0 patch merged23:39
kfox1111ah.23:39
kfox1111any successful gate runs after that?23:39
*** eaguilar has quit IRC23:40
*** eaguilar has joined #openstack-kolla23:40
*** lamt has quit IRC23:41
sbezverk_kfox1111: I think the gate was running ok after that..23:41
sbezverk_last night 6:35 the gate was ok23:41
sbezverk_https://review.openstack.org/44925423:41
openstackgerritKevin Fox proposed openstack/kolla-kubernetes master: WIP: Test gate  https://review.openstack.org/44984323:42
kfox1111sbezverk_: when you reviewed it, it was ok,23:42
kfox1111but it probably checked at 2:47 + ~30 min23:42
sbezverk_kfox1111: doubt filtering could be the cause as the gate was ok, but let's see23:43
kfox1111yeah, maybe not. just throwing a bunch of things against the wall to see what sticks. :)23:44
sbezverk_kfox1111: +123:44
*** lucasxu has joined #openstack-kolla23:45
sdakerwellum using the proper verson of  docker gets those doc sto work23:47
sbezverk_the moment of truth23:47
sdakerwellum i haen't sorted out how to start the api placement microchart or the cell0 stuff23:48
*** Pavo has quit IRC23:48
*** Pavo has joined #openstack-kolla23:49
*** schwicht has joined #openstack-kolla23:51
kfox1111sbezverk_: same failure.23:53
sbezverk_:(23:53
kfox1111your off the hook. ;)23:54
sbezverk_:)23:54
sbezverk_let's compare 6:35 log with the one we have now23:55
sbezverk_to see if we can spot any differences23:55
kfox1111ah. good plan.23:55
sbezverk_do you have any good tool to compare other than diff?23:56
kfox1111not really. :/23:57
kfox1111got a diff.. reviewing....23:59
sbezverk_kfox1111: yeah but timestamp screwed things up23:59

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!