Thursday, 2017-03-23

sbezverkkfox1111: no idea, but they do test starting vm at the gate too00:00
kfox1111I can believe it works. just wondering what unintended sideffects of running it without a cell1 are.00:00
sbezverkkfox1111: well it goes through all your basic tests00:01
kfox1111basic tests not nessisarily equate to no production problems.00:01
kfox1111stuff like this may only show up when upgrading from 4.x to 5.x.00:02
sbezverkI think it is a good start :), but agree about prod..00:02
kfox1111this is the kind of thing ops loose sleep over.00:02
sbezverkwell, adding cell1 later will take very little time if it is really needed00:02
kfox1111assuming it works though.00:03
kfox1111it may munge the db in a way thats not seperable.00:03
kfox1111or some cell0 cleaner task doesn't nuke all the instances cuase it doesn't expect live ones.00:03
kfox1111we really shoudl confirm with the nova folks cell0 only is an ok way to operate.00:04
sbezverkkfox1111: sure thing, I will post question on irc tomorrow when lots of nova folks around00:04
sbezverkkfox1111: or you can send question to mailer00:05
kfox1111yeah00:05
kfox1111sbezverk: reviewed00:07
kfox1111sbezverk: so, did you ever do a nova-manage cell_v2 discover_hosts ?00:07
sbezverkkfox1111: cool, thanks00:07
kfox1111hmm... I wonder if the simple setup does end up createing a cell1 too.00:08
kfox1111that coudl explain how it works.00:08
sbezverkkfox1111: nope, I think it is not needed since I made sure that compute nodes are up and running before creating cell000:08
kfox1111oh.00:09
sbezverkkfox1111: one nova dude told me so00:09
kfox1111uh.. is doing that inforced in the helm+kubernetes-entrypoint case?00:09
kfox1111I think we need a way to allow it to be done after the fact...00:09
kfox1111it sounds like you must rerun the process if you ever add more computes.00:09
sbezverkkfox1111: right dependency on nova-compute00:09
kfox1111but doesn't nova compute depend on nova-api?00:10
sbezverkkfox1111: yes00:10
kfox1111which shoudl depend on cells?00:10
sbezverkkfox1111: compute depends on api and cell depends on compute00:10
sbezverkapi does not depend on cell00:11
kfox1111I think the compute dep is oing to cause problems.00:12
kfox1111I think we should add a nova-manage cell_v2 discover_hosts job.00:12
kfox1111we will need it anyway.00:13
sbezverkkfox1111: your comment about nova.. it is actually how thet want00:14
sbezverkthey want user nova had access to nova_cell0 db00:14
sbezverkthere is no a separate user nova_cell000:14
kfox1111sbezverk: but the delete doesnt' use the same user as the create00:15
sbezverkkfox1111: right right  I have not noticed below config :)00:16
sbezverksorry00:16
kfox1111no worries. :)00:16
kfox1111so, simple_setup looks to be the migration tool.00:16
kfox1111so it may end up crating cell1 and moving stuff automatically.00:16
kfox1111though no idea what mariadb it uses. :/00:17
*** masber has quit IRC00:21
*** goldyfruit has quit IRC00:26
kfox1111sbezverk: would be curious to see the output of "nova-manage cell_v2 list_cells --verbose" on your cells setup.00:27
openstackgerritMohammed Naser proposed openstack/kolla master: Gate fixes: disable bifrost-base, fix ubuntu base image  https://review.openstack.org/44866600:27
*** tonanhngo has quit IRC00:33
*** adrian_otto has joined #openstack-kolla00:33
sdakekfox1111 sbezverk lets have a brainstorm session about HOW to do external configmaps if you have 30 minutes each plz00:34
kfox1111I gotta take off actually. maybe tomorrow?00:35
sdakekfox1111 sure ping sbezverk and me when you get in00:36
kfox1111kk.00:36
sdakekfox1111 i'd like to close on a design pronot :)00:36
sdakepronto that is00:36
*** tonanhngo has joined #openstack-kolla00:41
*** zhurong has joined #openstack-kolla00:41
*** duonghq has joined #openstack-kolla00:45
duonghqmorning guys00:45
*** tonanhngo has quit IRC00:45
*** tonanhngo has joined #openstack-kolla00:47
*** cuongnv has joined #openstack-kolla00:48
*** sayantani01 has quit IRC00:49
sdakesup duonghq00:51
sdakesteak tired00:51
openstackgerritJoseph Robinson proposed openstack/kolla-ansible master: [deploy-guide] Moving the quick start to the deploy folder  https://review.openstack.org/44850900:51
openstackgerritSerguei Bezverkhi proposed openstack/kolla-kubernetes master: Virtualbmc and deployment of baremetal VM using ironic  https://review.openstack.org/44155000:52
*** tonanhngo has quit IRC00:52
*** sayantani01 has joined #openstack-kolla00:52
*** tovin07_ has joined #openstack-kolla00:54
duonghqsdake, have we unblocked the gate?00:54
sdakeduonghq mnaser working on it i think00:54
sdakeduonghq but nope its busted00:54
mnaseri think00:54
mnasersdake00:54
mnaseri think i found it00:54
mnaserhttps://review.openstack.org/#/c/448666/00:55
mnaser3 fixes in total00:55
mnaseralso now the logs are MUCH more readable for failed images ^_^00:55
mnasernow we have logs/build which contain everything00:55
mnaserand the subunit problem doesnt poop out anymore00:55
duonghqmnaser, seem good00:56
duonghqyou combined your 2 ps?00:56
mnaseryeah00:56
sdakewhat does /tmp/logs/build do?00:56
sdakethe failed image logging is maddening to me00:57
mnasersdake http://logs.openstack.org/66/448666/4/check/gate-kolla-dsvm-build-ubuntu-binary-ubuntu-xenial-nv/beea00a/logs/build/ :)00:57
sdakei'm curious how its more readable now00:57
sdakecool hrw suggested that I think on the ml00:58
sdakehe will be happy to see it :)00:58
mnaseryeah the work was all done00:58
mnaseri didnt have to do much00:58
*** adrian_otto has quit IRC00:59
duonghqhttps://review.openstack.org/#/c/448666/3..4/tools/setup_gate.sh seem that we should wait for gate,01:00
duonghqhttp://logs.openstack.org/66/448666/3/check/gate-kolla-dsvm-build-centos-source-centos-7/e745bb1/console.html01:00
* mnaser is patiently watching the zuul status01:07
mnaser8 jobs left on this patch, all green so far01:07
duonghqnice01:08
mnaser7!01:10
duonghqmnaser, 601:10
mnaserill just feel a bit relieved with this in01:12
duonghqdo we provide note about disabling bifrost in doc?01:22
*** zhurong has quit IRC01:23
mnaserduonghq i think its more of a gate/testing thing, and not as much of an operational thing01:23
*** MasterOfBugs has quit IRC01:23
mnasersomeone has to look into fixing it, i tried my best but im a bit busy these next few days01:23
mnaserjust clearing out the gate took a lot into my $job work time and now i have to play catch up :(01:24
duonghqthank for your effort, gate is painful to debugging but must be done01:24
duonghqonly 4 left01:25
duonghqhope this will be ok01:25
mnaseryeah i think its good01:27
mnaseri love the new build logs01:27
mnaserthey're going to make life sooo much easier01:27
mnaseri dont know why we had it that way this whole time01:27
duonghqmnaser, you only add log_dir var and we have all nice log?01:27
mnaseryeah, it was already written01:27
duonghqah, yep01:28
duonghqnice feature but not used in gate yet,01:28
duonghqalso, I think it would be better if we put the line in separate commit, due to your commit has potential to be removed in future?01:29
duonghqhow do you think sdake01:29
*** masber has joined #openstack-kolla01:29
duonghqmnaser, seem that gate is all green now01:30
sdakeduonghq how not used in gate?01:31
duonghqsdake, I mean we should put logs_dir var in separate ps, due it isn't related to the bug but add nice output to our gate01:31
mnaserduonghq it actually breaks the gate01:31
mnaserif i dont add it there01:31
duonghqthe logs_dir var?01:32
*** japestinho has joined #openstack-kolla01:32
mnaserbecause the logs are so huge, python subunit complaints that the length of the payload is too big01:32
mnaserand it fails01:32
mnaserthat's why patch #3 failed01:32
duonghqhmm, got it01:32
mnaserif you look the log says "length too long" or something01:32
mnaserso i had no choice of doing it to unlock gate01:32
duonghqso we should note it somewhere01:32
duonghqnot sure01:32
duonghqor we put it in separate commit, and rebase the fix bug commit into it01:33
mnasernot possible01:33
mnaserbecause gate wont merge01:33
duonghqwhy?01:33
duonghqif parent of fix bug commit is the logs_dir commit01:33
mnaseryes but01:34
mnaserthe parent will never pass tests01:34
duonghqah, yeah01:34
mnaserbecause it is logging all errors to stdout/stderr01:34
mnaserso it will never merge01:34
duonghqcircular dep01:34
mnaseryep01:34
mnaser3 circular deps01:34
mnaserlol01:34
mnaserlog_dirs, netbase and bifrost disable01:34
duonghqpainful situation01:35
duonghqyour job is great01:35
duonghqone more test01:35
duonghq*your work01:36
duonghqmnaser, are you in US?01:37
mnasereast coast canada01:37
*** dave-mccowan has quit IRC01:37
duonghqnice01:37
mnaserat last01:39
mnasera +101:39
*** dave-mccowan has joined #openstack-kolla01:39
duonghqmnaser, done01:39
*** zhurong has joined #openstack-kolla01:43
*** manjeets has joined #openstack-kolla01:46
duonghqsbezverk, are you there?01:46
*** masber has quit IRC01:49
*** masber has joined #openstack-kolla01:53
*** hieulq has quit IRC01:56
*** fooliouno has quit IRC01:57
*** hieulq has joined #openstack-kolla01:59
*** dave-mcc_ has joined #openstack-kolla02:08
*** dave-mccowan has quit IRC02:10
*** goldyfruit has joined #openstack-kolla02:10
*** dave-mccowan has joined #openstack-kolla02:13
*** unicell has quit IRC02:15
*** dave-mcc_ has quit IRC02:16
*** eaguilar has quit IRC02:20
openstackgerritJoseph Robinson proposed openstack/kolla-ansible master: [deploy-guide] Moving the quick start to the deploy folder  https://review.openstack.org/44850902:28
*** fooliouno has joined #openstack-kolla02:39
openstackgerritMerged openstack/kolla master: Gate fixes: disable bifrost-base, fix ubuntu base image  https://review.openstack.org/44866602:39
*** hieulq has quit IRC02:39
*** hieulq has joined #openstack-kolla02:41
*** l4yerffeJ has joined #openstack-kolla02:41
*** manheim has joined #openstack-kolla02:45
*** unicell has joined #openstack-kolla02:48
*** manheim has quit IRC02:50
*** 7IZAAUKJJ has joined #openstack-kolla02:52
*** masber has quit IRC02:56
*** 7IZAAUKJJ has quit IRC03:00
openstackgerritSayantani Goswami proposed openstack/kolla-ansible master: [deploy-guide] Moving the quick start to the deploy folder  https://review.openstack.org/44850903:03
openstackgerritOpenStack Proposal Bot proposed openstack/kolla master: Updated from global requirements  https://review.openstack.org/44677103:05
openstackgerritZeyu Zhu proposed openstack/kolla-ansible master: Update the kubernetes-deployment.rst file  https://review.openstack.org/44803503:08
*** l4yerffeJ has quit IRC03:10
*** eaguilar has joined #openstack-kolla03:11
*** kbaegis has quit IRC03:13
*** zhubingbing has joined #openstack-kolla03:17
openstackgerritSerguei Bezverkhi proposed openstack/kolla-kubernetes master: Virtualbmc and deployment of baremetal VM using ironic  https://review.openstack.org/44155003:24
spsuryahello all03:26
duonghqhi spsurya03:33
openstackgerritDuong Ha-Quang proposed openstack/kolla master: Implement Keystone zero-downtime upgrade  https://review.openstack.org/42583303:33
*** bot_Jeffrey4l has joined #openstack-kolla03:38
duonghqsbezverk, ping03:41
*** bot_Jeffrey4l has quit IRC03:42
*** bot_Jeffrey4l has joined #openstack-kolla03:43
*** eaguilar has quit IRC03:43
sbezverkduonghq: hey what is up?03:46
*** bot_Jeffrey4l has quit IRC03:47
*** bot_Jeffrey4l has joined #openstack-kolla03:47
*** bot_Jeffrey4l has quit IRC03:48
*** zhurong has quit IRC03:48
duonghqsbezverk, about the ps: https://review.openstack.org/#/c/448258/803:48
*** masber has joined #openstack-kolla03:48
duonghqI mean tool for generate openrc file, the kube already has it?03:49
*** dave-mccowan has quit IRC03:50
*** hieulq has quit IRC03:50
*** hieulq has joined #openstack-kolla03:53
openstackgerritSurya Prakash (spsurya) proposed openstack/kolla master: Fix oslo_debug_helper not running for kolla/tests directory  https://review.openstack.org/44439803:56
duonghqsbezverk, what is vip are you using?03:59
duonghq(compare to nodes' ip)03:59
openstackgerritSerguei Bezverkhi proposed openstack/kolla-kubernetes master: Virtualbmc and deployment of baremetal VM using ironic  https://review.openstack.org/44155003:59
sbezverkduonghq: vip is public ip assigned to host running kube cluster04:00
duonghqsbezverk, what if I run in multinode setup?04:00
duonghqfor local test04:01
duonghqand why dns_name of kolla is same as external_vip?04:01
sbezverkduonghq: then it could be ANY publically available ip04:02
sbezverkused by your cluster04:02
duonghqand about dns_name, the kubedns is not expose in my cluster, does it matter?04:03
sbezverkduonghq: you have to have kubedns running otherwise kolla-kube will not work04:05
duonghqsbezverk, kubedns is runnning but donot have external ip04:05
duonghqonly cluster ip04:05
sbezverkthat is normal04:06
duonghqand it's ok?04:06
sbezverkdsn_name has no realtion to kubedns04:06
duonghqI got trouble with network again04:06
duonghqEntrypoint WARNING: 2017/03/23 04:07:39 entrypoint.go:81: Resolving dependency for mariadb failed: Get https://10.233.0.1:443/api/v1/namespaces/kolla/endpoints/mariadb: dial tcp 10.233.0.1:443: getsockopt: no route to host04:07
duonghq04:07
duonghqit's output of one of job04:07
sbezverkdns_name was introduced just for the case when you have external dns mapped04:07
sbezverkto one of ip addresses used by kube cluster04:07
duonghqah, got it04:07
sbezverkso normally they are the same04:08
sbezverkvip and dns name but could be different04:08
sbezverkif it is needed04:08
sbezverkduonghq: got to go, it is very late now.. talk to you tomorrow04:08
duonghqsbezverk, thank you, cya04:09
fooliounoduonghq: do you have a multi node kolla-k8s cluster up?04:09
duonghqfooliouno, yes04:10
duonghqah, no04:10
duonghqI only have multinode k8s04:10
*** fooliouno_ has joined #openstack-kolla04:10
fooliouno_sorry, got disconnected04:11
fooliouno_do you have it up04:11
duonghqfooliouno_, I have only k8s cluster04:12
duonghqbut not kolla-k8s04:12
fooliouno_flannel/weave/canal?04:12
duonghqyup04:12
fooliouno_I cant get flannel overlay to work on k8s. Cant get keystone_create_db to resolve mariadb host address04:13
*** fooliouno has quit IRC04:14
fooliouno_did you run into any issues with resolving service names within containers04:14
duonghqfooliouno_, I just got it error today04:16
duonghqfooliouno_, how do you setup k8s?04:16
fooliouno_kubeadm04:16
fooliouno_hosted install04:16
duonghqhmm, I used kargo04:16
duonghqcan you check kubeproxy log04:17
fooliouno_what should I be looking for04:17
*** goldyfruit has quit IRC04:18
duonghqcheck if you have any iptables-related warning04:18
fooliouno_ok .. let me take a look04:19
*** tovin07_ has quit IRC04:25
fooliouno_duonghq: what version of docker do you have04:27
duonghqfooliouno_, Server Version: 1.13.104:28
fooliouno_ok .. I have the same. Just want to rule out any differences :)04:28
fooliouno_Were you able to resolve the error with mariadb04:29
*** manheim has joined #openstack-kolla04:30
*** yee379 has quit IRC04:30
*** yee37914 has joined #openstack-kolla04:31
duonghqI was able, but after some restart the VMs run k8s cluster, somehow it's broken04:31
duonghqI'm reinstall cluster04:31
*** manheim has quit IRC04:34
*** jrobinson has quit IRC04:36
*** tovin07_ has joined #openstack-kolla04:36
*** tovin07_ has quit IRC04:38
*** masber has quit IRC04:40
duonghqfooliouno_,04:41
duonghqI can createdb04:41
fooliouno_keystone db?04:41
duonghqyes04:42
fooliouno_thats good! I'm still trying to figure out whats wrong with mine :)04:42
duonghqcan you provide kubectl get po -n kolla04:43
fooliouno_Do you know of anyone who has gone through the entire kolla-k8s setup and created nova instances?04:43
duonghqseem that no one online now04:43
fooliouno_I just did a kubeadm reset to reset the cluster and start from scratch04:43
*** tovin07_ has joined #openstack-kolla04:44
*** MasterOfBugs has joined #openstack-kolla04:44
*** Jeffrey4l has joined #openstack-kolla04:44
fooliouno_duonghq: one other question regarding the container images. Do you build them from ubuntu source. what is your build date04:44
duonghqI use centos binary image04:45
duonghq3 weeks ago04:46
fooliouno_Ah . .ok. And do you use the kolla-kubernetes commands or helm04:46
duonghqI use helm04:47
duonghqkubectl04:47
duonghqdue to kolla-kubernetes is not usable yet04:47
fooliouno_But I am using the commands on the kolla-k8s page and they seem to bring up the containers well. Is there a link for the helm commands04:47
duonghqfooliouno_, some here: https://review.openstack.org/#/c/447356/04:49
fooliouno_duonghq: This is what I am using: http://pastebin.com/Yw1dEanS04:49
sdakefooliouno_ i have done so04:50
fooliouno_sdake: baremetal, ubuntu 16.04, kubeadm, flannel is what I have. Is yours similar04:51
duonghqI still have trouble about keystone cannot connect to keystone-internal04:51
sdakefooliouno_ video of the work: https://www.youtube.com/watch?v=rHCCUP2odd804:51
sdakefooliouno_ in my setup I use centos04:52
fooliouno_duonghq: I think I saw that error as well when my cluster was up.04:52
duonghqfooliouno_, ya04:52
sdakefooliouno_ we are working on a deployment guide here: https://review.openstack.org/#/c/447356/04:53
sdakeduonghq you know i recall that error04:53
sdakeduonghq let me hunt in my notes and see if i have a referene to how i fixed it04:53
fooliouno_sdake: I have been trying to get kolla-k8s to work using the kolla-kube commands on the quickstart guide pages for a few weeks now. Let me watch your video. Thanks!04:53
duonghqsdake,  I resoled the resolve-conf04:53
sdakeduonghq right - i have a different solution then that04:54
*** Jeffrey4l has left #openstack-kolla04:55
sdakefooliouno_ the quickstart guide pages are woefully out of date04:55
sdakefooliouno_ hence why we are working on a proper deployment guide04:55
*** Jeffrey4l_ has quit IRC04:55
sdakeduonghq can you run openstack catalog list04:56
sdakeduonghq or the command that is similar to that04:56
duonghqopenstack user list is ok?04:56
sdakeduonghq no, must be catalog list04:56
fooliouno_sdake: unfortunately, thats what I have been struggling with. So, if I follow your new deployment guide all the way, I should be able to create nova instances as well in a multi node environment?04:56
duonghqyeah04:56
sdakeduonghq yeah that works ?04:56
*** Jeffrey4l has joined #openstack-kolla04:56
duonghqhttp://paste.openstack.org/show/603852/04:57
duonghqyes04:57
sdakeduonghq winning - i have a solution for your problem04:57
sdakefooliouno_ if you can understand the guide as it stands today - it needs editing :)04:57
duonghqsdake, you don't say I should set dns_name?04:57
sdakeduonghq the issue has nothign to do with dns04:58
sdakekeystone 35357 is not externally routed04:58
sdakeany command that requires administrative privileges will not work (such as user list, endpoint list etc)04:58
sdakeduonghq there is a solution to this04:58
sdakefooliouno_ we are working on fixing the deploy guide and hope to have it published this week04:59
sdake(note iso standard says weeks start on sunday :)04:59
fooliouno_sdake: that would be a life saver for me. Been struggling with it for too long to admit :)04:59
sdakefooliouno_ right -the docs are terrible04:59
sdakeall across kolla unfortunately04:59
sdakefooliouno_ we don't have a professional documentation writer in kolla at this point in time that i am aware of05:00
sdakedoesn't mean there aren't people doing that :)05:00
sdakefooliouno_ i feel your pain re struggling it took me 2 weeks to get kolla-kubernetes up and running and documented in a basic state in review https://review.openstack.org/#/c/447356/05:01
fooliouno_sdake: Let me start with that guide tomorrow. I'll be more engaged on this channel hereafter :)05:02
duonghqfooliouno_, which tz are you in?05:02
sdakefooliouno_ nice - tonight i'm going to fix up that deploy guide a bit so its more readable05:02
fooliouno_Central05:02
sdakeMST here05:02
*** shashank_t_ has joined #openstack-kolla05:03
duonghqsdake, [OT] how about the weather at your place these days?05:03
sdakeduonghq i'm good with ot discussion05:04
sdakeits actually cold atm05:04
duonghqhmm, nice05:04
sdake17C05:04
fooliouno_duonghq: sdake: Thanks for your help. I'm gonna call it a night. Will be back tomorrow.05:04
duonghqfooliouno_, night05:04
sdakeduonghq although during the days its 90F which is pretyt warm05:04
sdakefooliouno_ enjoy - hope the video was helpful :)05:05
sdakefooliouno_ it can be done :)05:05
duonghqsdake,  same here05:05
sdakeso duonghq now that I understand what your problem is05:06
sdakeduonghq let me sort out what was done to fix it05:06
duonghqroger05:06
sdakeduonghq how did you generate your cloud.yaml?05:06
duonghqsdake, some script in project, then modify as common_workflow_config.05:08
sdakeduonghq i see05:08
duonghqsdake, tools/build_example_yaml.py05:08
sdakeduonghq you need to override... something05:09
sdakemoment05:09
*** jrobinson has joined #openstack-kolla05:14
duonghqsdake, here is my cloud.yaml: http://paste.openstack.org/show/603854/05:23
sdake[22:15:50]  <sdake>duonghq http://paste.openstack.org/show/603853/05:24
sdake[22:16:13]  <sdake>duonghq you need line 3505:24
sdake[22:16:30]  <sdake>duonghq that defaults to false for whatever reason05:24
sdake[22:16:49]  <sdake>you need it under the global.kolla.all namespace05:24
*** cuongnv has quit IRC05:25
duonghqsdake, I already set it to true for keystone05:25
sdakeduonghq i dont see it in your cloud.yaml you pasted05:26
duonghqsdake, line 16605:26
sdakeduonghq it needs to be in the global.kolla.all section not in the keystone.all section05:27
duonghqsdake, ok05:27
sdakedelete line 166 while your at it :)05:27
duonghqI only need to delete keystone chart and install again?05:28
sdakehelm delete compute-kit --purge05:28
sdakehelm nstall compute-kit05:28
duonghqcan I just install keystone05:28
sdakeyou can try and report back :)05:28
duonghqya05:28
duonghqI need a quick nap, cya05:29
duonghqthank you05:29
*** cuongnv has joined #openstack-kolla05:32
sdakenp05:33
spsuryasayantani01: ping...05:42
sayantani01spsurya Hi :)05:42
spsuryasayantani01:  can we add some measures reagdring gates issue within this patch https://review.openstack.org/#/c/436462/05:43
spsurya?05:43
spsuryasayantani01: though all looks good05:44
spsuryabut bifrost gate failure a big pain05:44
sayantani01Since this patch deals with only documentation, wouldn't it be better to take up the gate issue in a separate patch?05:45
*** skramaja has joined #openstack-kolla05:46
sayantani01spsurya, I am about to go to bed. I will take another look tomorrow :)05:46
spsuryasayantani01: sure05:47
*** bmace has quit IRC05:47
*** sayantani01 has quit IRC05:48
*** bmace has joined #openstack-kolla05:48
*** zhurong has joined #openstack-kolla05:53
*** jascott1 has quit IRC05:54
*** jascott1 has joined #openstack-kolla05:55
*** shashank_t_ has quit IRC05:57
*** jascott1 has quit IRC05:59
zhubingbingdough06:01
zhubingbingsup sdake suoghq06:01
zhubingbingduonghq06:01
zhubingbing;)06:01
zhubingbingi know kolla-k8s dns problems06:01
zhubingbing;)06:01
zhubingbingi fix it06:01
zhubingbing;)06:01
*** qiliang28 has joined #openstack-kolla06:04
*** haplo37_ has joined #openstack-kolla06:06
sdakehey zhubingbing06:06
zhubingbinghi sdake06:06
*** tonyb_ has joined #openstack-kolla06:06
duonghqsdake, I'm installing compute-kit06:07
duonghqhope that it'll be fine06:07
sdakeduonghq sounds good06:08
sdakeworking on docs now06:08
sdakeneed testing soon :)06:08
duonghqroger06:08
*** bpetit has quit IRC06:08
*** bpetit has joined #openstack-kolla06:08
*** kbyrne_ has joined #openstack-kolla06:09
duonghqhmm, got iscsi crashbackloo06:09
duonghqcrashloopbackoff06:09
*** unicell1 has joined #openstack-kolla06:10
duonghqbut what is 3.0.3-beta.1? it's from ocata?06:10
*** unicell has quit IRC06:11
*** jrobinson has quit IRC06:11
*** tonyb has quit IRC06:13
*** flaper87 has quit IRC06:13
*** haplo37 has quit IRC06:13
*** kbyrne has quit IRC06:13
*** qiliang27 has quit IRC06:13
*** haplo37_ is now known as haplo3706:13
*** kbyrne_ is now known as kbyrne06:13
*** qiliang28 is now known as qiliang2706:13
duonghqsdake, not sure why I get this potion of service list:06:13
duonghqkeystone-admin      10.233.27.189   <none>            35357/TCP   9m06:13
duonghqkeystone-internal   10.233.55.70    192.168.122.248   5000/TCP    9m06:13
duonghqkeystone-public     10.233.22.176   192.168.122.248   5000/TCP    9m06:13
sdakeduonghq its pre-ocata06:13
sdakeduonghq its acutally 3.0.3 pre-release06:13
duonghqsdake, it's from dockerhub?06:13
sdakeduonghq yes06:13
duonghqI'm using my local registry, hope that it will be fine06:14
*** jrobinson has joined #openstack-kolla06:15
duonghqhmm, I get keystone-manage-db-prwnp                           1/1       Running            0          12m06:15
duonghq but keystone blames that Entrypoint WARNING: 2017/03/23 06:17:56 entrypoint.go:81: Resolving dependency for keystone-manage-db failed: Job keystone-manage-db is not completed yet06:15
duonghqah still in migration process06:16
Jeffrey4lls06:17
zhubingbinghttp://paste.openstack.org/show/603857/06:18
zhubingbingsdake duoghq06:18
openstackgerritSteven Dake proposed openstack/kolla-kubernetes master: WIP: Deployment Guide Documentation  https://review.openstack.org/44735606:18
zhubingbingcan u look it06:18
sdakezhubingbing i did look06:19
sdakezhubingbing i'm a bit winded (11pm here) - not sure what to make of it :)06:19
zhubingbingwrite a mess06:20
zhubingbingduonghq he know it06:20
duonghqzhubingbing, so we should match the kubeadm api with machine ip,06:21
duonghqweired, it doesn't describe in the official doc,06:21
duonghqiirc06:21
*** masber has joined #openstack-kolla06:22
zhubingbing;)06:22
sdakezhubingbing makes up his own docs :)06:23
duonghqsdake, I don't know why but with admin_port_external: true the keystone-admin doesn't get external ip06:24
sdakeduonghq paste your cloud.yaml06:25
duonghqhttp://paste.openstack.org/show/603858/06:25
*** tovin07_ has quit IRC06:26
zhubingbingdns_name: 192.168.122.24806:28
zhubingbingdon;t use it06:28
sdakeduonghq http://paste.openstack.org/show/603859/06:29
sdakeduonghq set that for your keystone section06:29
zhubingbingyeah sdake +106:30
duonghqsdake, so global.kolla.all doesn't set for kolla.keystone.all?06:30
*** jaosorior has joined #openstack-kolla06:30
sdakeduonghq kolla.keystone.all overrides anything in globals.kolla.all06:30
sdakecounterintuitive i know06:31
duonghqI think it's ok06:31
duonghqbut I do not set anything in kolla.keystone.all06:31
duonghqor the default value of its overrides globals one?06:31
sdakeyes you need to override the defaults06:31
sdakejust cut and paste it at line 16206:32
duonghqit'll surprising somebody06:32
sdake(in your cloud.ayml)06:32
duonghqya, I'm deleting old compute-kti06:32
zhubingbingDo you think  to record a video with how to deploy kolla-k8s06:33
zhubingbing;)06:33
duonghqbut even three keystone service got external ip, it still cannot contact to keystone-internal iirc06:33
zhubingbinghttps://review.openstack.org/44735606:33
zhubingbinguse this guide documentation06:33
sdakeduonghq do you have a working external ip for keystone?06:34
duonghqfor keystone-admin and keystone-public: yes,06:34
sdakeduonghq keystone endpoint list working?06:35
duonghqnot trying this yet06:35
duonghqmins, I need to bring up it again06:35
sdakezhubingbing are you using 101 atm?06:35
duonghqhelm delete --purge compute-kits takes long time06:35
sdakeduonghq about 8 mins on my gear ;)06:35
zhubingbingem yes sdake06:36
sdakezhubingbing how are you using it - i want to setup kubeadm on it and run through the documentation06:36
sdakezhubingbing to test it and see if it works06:36
zhubingbingok06:36
duonghqso it will be worse on my mini PC06:36
sdakezhubingbing better yet, you can test it if you like06:36
zhubingbingno prolemes i will test in06:36
sdakealthough for some reason 102 and 103 are not coming up06:37
openstackgerritLingyong Xu proposed openstack/kolla master: Using assertIsNone() instead of assertEqual(None)  https://review.openstack.org/44892306:37
sdakezhubingbing i'm going to go look at 102 and 103 and see what the eal is with it06:37
zhubingbingok06:37
openstackgerritLingyong Xu proposed openstack/kolla master: Using assertIsNone() instead of assertEqual(None)  https://review.openstack.org/44892306:39
sdakezhubingbing can you try out 101 and taint it so you can run on just 10106:39
*** masber has quit IRC06:39
openstackgerritjianyi proposed openstack/kolla-ansible master: Add zun ansible role  https://review.openstack.org/41617906:40
zhubingbinghmm i can ,i am use halcyon to set my dev env06:40
sdakezhubingbing i see, I want to run through the instructions in the deployment guide - on bare metal06:41
openstackgerritZeyu Zhu proposed openstack/kolla-ansible master: Modify the permission of directory  https://review.openstack.org/44893206:42
zhubingbinghmm yeah06:42
sdakezhubingbing the review i just pushed - so I'll do that now06:42
zhubingbingi will fix the gate , let the same as the deployment guide06:43
*** shashank_t_ has joined #openstack-kolla06:47
*** satyar has joined #openstack-kolla06:48
*** rwsu has joined #openstack-kolla06:48
duonghqsdake, if I set the keystone to your snippet, the keystone-internal does not get external ip06:48
sdakeduonghq what about keystone external ?06:49
sdakekeystone admin/06:49
sdakei mean06:49
*** shashank_t_ has quit IRC06:50
duonghqsdake,  here06:50
duonghqkeystone-admin      10.233.14.21    192.168.122.248   35357/TCP   2m06:50
duonghqkeystone-internal   10.233.36.85    <none>            5000/TCP    2m06:50
duonghqkeystone-public     10.233.57.160   192.168.122.248   5000/TCP    2m06:50
sdakewell thats progress :)06:50
duonghqopenstack endpoint list still blame: Failed to contact the endpoint at http://keystone-internal:5000/v3 for discovery. Fallback to using that endpoint as the base url.06:51
sdakestill progress06:51
duonghqyeah, I got this error again06:51
sdakeprior to adding the snippet did keystone-admin have an external ip?06:51
duonghqI got ip for keystone-admin and public in place before06:51
duonghqI managed to have it sometime06:51
duonghqbut w/o the snippet, it can not have the ip, absolutely06:52
sdakeit being which object?06:52
duonghqsorry?06:52
sdakewithout the snipped what does nothave the external ip?06:53
sdakei provided two snippets want to know which ones triggered which behavior06:53
duonghqhmm, the keystone snippet enable keystone-admin ip, but not keystone-internal06:53
sdakeok06:54
duonghqthe other does not much thing06:54
duonghq(it my case)06:54
sdakeprior to the snippet keystone-internla was enabled correct?06:54
duonghqiirc, yes06:55
duonghqhttp://paste.openstack.org/show/603861/ the current error06:55
sdakeset this to true:         node_port_enabled: false06:55
duonghqsdake, Error: release keystone failed: Service "keystone-public" is invalid: spec.ports[0].nodePort: Invalid value: 5000: provided port is not in the valid range. The range of valid ports is 30000-3276706:57
sdakezhubingbing any ideas?06:58
duonghqgot I before, so I left it disable06:59
zhubingbingi look it06:59
zhubingbingwait 1min06:59
zhubingbingi thinks keystone craete endpoint have some problems07:00
*** tovin07_ has joined #openstack-kolla07:01
*** sbezverk_ has joined #openstack-kolla07:01
*** targon has joined #openstack-kolla07:01
zhubingbingIf i can log in to the machine operation just fine07:02
zhubingbingduonghq07:02
zhubingbing;)07:02
duonghqit's not possible here,07:02
duonghqquite sad07:02
*** dmellado has quit IRC07:03
duonghqzhubingbing, do you have network setting in systemd?07:03
duonghqcalico...07:03
zhubingbinghmm07:03
zhubingbingAyesha07:03
zhubingbingi use flnall07:03
*** bpetit has quit IRC07:03
zhubingbingflanel07:03
zhubingbing;)but i think07:04
zhubingbingyou can attach public the machetes ?07:04
*** kfox1111 has quit IRC07:04
*** kfox1111 has joined #openstack-kolla07:04
*** sbezverk has quit IRC07:05
*** bpetit has joined #openstack-kolla07:05
duonghqzhubingbing, sorry?07:05
sdakeduonghq try pinging bradjones or sbezverk_ when they appear07:08
*** dmellado has joined #openstack-kolla07:08
sdakei have to hit the rack07:08
duonghqsdake, roger07:08
sdakezhubingbing duonghq if either or both of you could go throug hthe deploy guide that would be good :)07:08
sdakeand provide a review07:08
sdakefix mistakes/etc07:08
sdake(in comments or by updating the review directly07:09
duonghqgot it07:09
zhubingbinghmm07:09
zhubingbingok07:09
sdakeup since 5am - need atleast 6-7 hrs a night of sleep :)07:09
sdakeenjoy07:09
*** jmccarthy has joined #openstack-kolla07:09
zhubingbing;)07:09
*** jrobinson has quit IRC07:11
duonghqnight sdake07:17
jaosoriormandre: hey, could you check this out https://review.openstack.org/#/c/446911/ ?07:24
mandrejaosorior: looks like you did it :)07:26
mandrejaosorior: what about commenting out the 'Listen 443 https' line intead of deleting it, to be more consistent with the change made to httpd.conf for port 80?07:29
*** cuongnv has quit IRC07:29
jaosoriormandre: sure07:29
mandrejaosorior: it's not a big deal though07:30
jaosoriormandre: I'll do that, it'll be in a couple of hours though, since I'm finishing up some other task07:32
*** manheim has joined #openstack-kolla07:33
*** manheim has quit IRC07:36
*** manheim has joined #openstack-kolla07:36
*** mnasiadka has joined #openstack-kolla07:39
*** Serlex has joined #openstack-kolla07:52
*** matrohon has joined #openstack-kolla07:53
openstackgerritJun Hu proposed openstack/kolla master: Fix ironic-donductor failed when build with binary  https://review.openstack.org/44897707:55
openstackgerritSurya Prakash (spsurya) proposed openstack/kolla master: Fix oslo_debug_helper not running for kolla/tests directory  https://review.openstack.org/44439807:55
openstackgerritMerged openstack/kolla master: Add libosinfo package for source installation  https://review.openstack.org/44570107:56
openstackgerritJun Hu proposed openstack/kolla master: Fix ironic-donductor failed when build with binary  https://review.openstack.org/44897707:57
*** cuongnv has joined #openstack-kolla07:58
spsuryamandre:  https://review.openstack.org/444398   your commnets has been incorporated and some addition information for your query07:59
openstackgerritJun Hu proposed openstack/kolla master: Fix ironic-donductor failed when build with binary  https://review.openstack.org/44897708:03
*** rmart04 has joined #openstack-kolla08:03
mandrejaosorior: I noticed we forgot to disable port 80 for a few images, I'm going to push a patch and have yours depend on it... I'll also edit your patch to comment out the line08:04
*** shardy has joined #openstack-kolla08:05
jaosoriormandre: oh, that would be great dude, thanks08:05
*** pcaruana has joined #openstack-kolla08:06
*** flaper87 has joined #openstack-kolla08:06
*** flaper87 has quit IRC08:07
*** flaper87 has joined #openstack-kolla08:07
*** rmart04 has quit IRC08:07
mandrespsurya: the thing is, I didn't see a difference with explicitely listing directories or not08:12
spsuryamandre: after adding the kolla/tests. We are able to run all the remaining tests08:13
spsuryaaround 30 cases are in tests and 27 are in kolla/tests08:14
spsuryaso now we are able to run 57 cases08:15
mandrespsurya: "tox -e debug" runs the same number of tests for me with and without your patch, unless I'm missing something08:15
*** mkoderer has joined #openstack-kolla08:17
mandrespsurya: if I switch 'commands' to 'commands1', it does run all the 57 tests, and also much faster08:18
mandrespsurya: idk what commands1 stands for08:18
*** matrohon has quit IRC08:20
*** Administrator_ has quit IRC08:20
spsuryamandre: i have seen somewhr before that is why i change that ..... but after your comment i reverted08:21
*** manheim has quit IRC08:21
*** Administrator_ has joined #openstack-kolla08:21
spsuryaso eaxctly not much idea but it works08:21
*** manheim has joined #openstack-kolla08:21
spsuryamandre: and yes "tox -e debug" doesn't run all the cases08:21
spsuryait runs only 30 If i see correclty right now08:22
openstackgerritSurya Prakash (spsurya) proposed openstack/kolla master: Fix oslo_debug_helper not running for kolla/tests directory  https://review.openstack.org/44439808:23
spsuryamandre:   with "commands1"  below is the logs where we can see the difference w.r.t "commands" only08:27
spsuryahttps://www.irccloud.com/pastebin/QYHrhMCJ/08:27
spsuryahttps://www.irccloud.com/pastebin/dmfOTlmh/08:28
spsuryamandre: please see you got something of it08:28
*** rmart04 has joined #openstack-kolla08:30
*** matrohon has joined #openstack-kolla08:35
*** rmart04 has quit IRC08:35
*** egonzalez has joined #openstack-kolla08:44
*** bjolo has joined #openstack-kolla08:45
bjolomorning08:45
duonghqmorning bjolo08:45
duonghqmorning eglute08:45
duonghqegonzalez,08:45
bjolohi duonghq08:45
duonghqsorry eglute08:45
bjolohows vietnam?08:46
egonzalezmorning08:46
duonghqbjolo, weather is becoming summer, quite hot now, it'll be colder in weekend (with rain)08:46
hawihi. question - is there some way to make Horizon running quicker for multiple users? as i undestand Horizon is just one container, yes?08:47
*** mgoddard has joined #openstack-kolla08:47
duonghqegonzalez, if you have spare time, can you take a look at this: https://review.openstack.org/#/c/425833/ iirc, the ubuntu-binary gate red due to fixed bug08:47
egonzalezduonghq, error in ubuntu in this PS is unrelated tho, rechecked to ensure08:48
duonghqegonzalez, ara error was fixed once08:49
spsuryabjolo: morning08:49
duonghqI'm not sure why it still appear here08:50
spsuryaduonghq:  hi...08:50
egonzalezhawi, for running quicker what you mean? if you have a lot of users, have to scale horizon container with more nodes08:50
egonzalezduonghq, the error was about mariadb not connecting. is not the same08:50
spsuryamandre: ^^08:50
*** jaosorior is now known as jaosorior_brb08:55
*** qiliang28 has joined #openstack-kolla08:56
openstackgerritMerged openstack/kolla master: Update to Kubernetes 1.5.4  https://review.openstack.org/44721708:58
*** shardy has quit IRC08:58
*** qiliang27 has quit IRC08:59
*** qiliang28 is now known as qiliang2708:59
*** shardy has joined #openstack-kolla09:00
*** openstackgerrit has quit IRC09:03
*** athomas has joined #openstack-kolla09:03
*** openstackgerrit has joined #openstack-kolla09:03
openstackgerritMerged openstack/kolla-ansible master: Improved documentation for central logging.  https://review.openstack.org/44447709:03
hrwmnaser: THANK YOU!09:09
hrwsdake: thanks too - would not noticed without you09:10
*** david-lyle has quit IRC09:11
*** jrist has quit IRC09:12
*** david-lyle has joined #openstack-kolla09:13
*** jrist has joined #openstack-kolla09:13
spsuryahttps://www.irccloud.com/pastebin/8MpW4q6I/09:13
spsuryaegonzalez:  what is the exact difference between commands and commands1, if we use in tox.ini debug environment09:13
spsuryaplease take a look on above log09:14
egonzalezpbourke, is this a bug or is expected to have build logs from mitaka in master? https://github.com/openstack/kolla/blob/master/docker/base/Dockerfile.j2#L14809:15
*** cu5 has quit IRC09:22
*** kbaegis has joined #openstack-kolla09:22
egonzalezspsurya, to be fair I don't know. Cannot find any reference for command1 in other tox files09:25
*** david-lyle_ has joined #openstack-kolla09:27
egonzalezspsurya, I think is a typo, if you look at the other changes made to the same bug, none include command1, only add the correct tests path09:27
*** david-lyle has quit IRC09:27
*** cu5 has joined #openstack-kolla09:28
*** cu5 has quit IRC09:28
spsuryaegonzalez: but don't know why it is running 57 even take less time to execute :(09:29
spsuryayeh it is typo09:29
egonzaleztry to execute it verbose and see what tests are made09:29
*** fooliouno_ has quit IRC09:32
*** manheim has quit IRC09:37
*** manheim has joined #openstack-kolla09:37
spsuryaegonzalez: sorry09:42
spsuryabut i have not idea how to execute in verbose09:42
*** matrohon has quit IRC09:42
*** matrohon has joined #openstack-kolla09:43
*** rmart04 has joined #openstack-kolla09:43
egonzalezspsurya, tox -e (test_to_execute) --debug09:44
egonzalezspsurya, nope, --debug does not exist , sorry09:44
spsuryaegonzalez: roger09:45
spsuryai think only debug would be ok09:45
spsuryafor dingle test case to run09:45
spsuryasingle*09:46
*** rmart04 has quit IRC09:47
egonzalezspsurya, can set an output json file with detailed info --result-json /tmp/result.json09:48
spsuryaegonzalez: roger09:51
spsuryaegonzalez:  http://paste.openstack.org/show/603878/   but seems like getting nothing of it09:56
*** duonghq has quit IRC09:59
*** MasterOfBugs has quit IRC09:59
*** satyar has quit IRC10:00
egonzalezspsurya, with command1 is not running all tests http://paste.openstack.org/show/603879/10:01
spsuryaegonzalez: you mean commands1 not running any case10:03
egonzalezspsurya, yep10:03
spsuryaegonzalez: updated on same     http://paste.openstack.org/show/603880/10:07
spsuryaso why the output of  ``tox -e debug`` showing 57 tests passed ?10:08
spsuryaon stdout10:08
openstackgerritMartin André proposed openstack/kolla master: RHEL: Add mod_ssl for services running over httpd  https://review.openstack.org/44691110:08
openstackgerritMartin André proposed openstack/kolla master: Prevent apache from listening on port 80  https://review.openstack.org/44903910:08
openstackgerritMartin André proposed openstack/kolla master: Prevent apache from listening on port 80 for heat-base  https://review.openstack.org/44904010:08
*** kbaegis has quit IRC10:11
egonzalezspsurya, replied http://paste.openstack.org/show/603883/10:13
egonzalezspsurya, as it is a wrong command statement, is not using the content of the command110:14
*** pbourke has quit IRC10:14
*** pbourke has joined #openstack-kolla10:16
*** kbaegis has joined #openstack-kolla10:16
*** cuongnv has quit IRC10:18
*** kbaegis has left #openstack-kolla10:18
spsuryaegonzalez:   got that command1 content is not being used as per --showconfig10:19
egonzalezyep10:19
*** rmart04 has joined #openstack-kolla10:21
spsuryaegonzalez: sorry to say but the only thing is bugging me is ``tox -e debug`` output10:22
spsurya:(10:22
spsuryathnks for getting the command1 content10:23
*** rmart04 has quit IRC10:25
hrwomg... jenkins gave +1 to my patch...10:33
spsuryaegonzalez:  from these logs it seems it running all the slowest tests10:34
spsuryaand skiping the third argument10:34
*** manheim has quit IRC10:35
openstackgerritSurya Prakash (spsurya) proposed openstack/kolla master: Fix oslo_debug_helper not running for kolla/tests directory  https://review.openstack.org/44439810:37
*** athomas has quit IRC10:41
*** rabel has joined #openstack-kolla10:49
*** zhubingbing has quit IRC10:49
*** rhallisey has joined #openstack-kolla10:58
*** jaosorior_brb is now known as jaosorior11:02
openstackgerritMarcin Juszkiewicz proposed openstack/kolla master: ironic-pxe: handle non-x86 architectures  https://review.openstack.org/43481711:03
openstackgerritMarcin Juszkiewicz proposed openstack/kolla master: debian: enable all images enabled for Ubuntu  https://review.openstack.org/43278711:03
openstackgerritMarcin Juszkiewicz proposed openstack/kolla master: Add support for non-x86 architectures (aarch64, ppc64le)  https://review.openstack.org/43094011:03
openstackgerritMarcin Juszkiewicz proposed openstack/kolla master: openstack-base/kolla-toolbox: use mariadb-devel for CentOS and !x86-64  https://review.openstack.org/44483211:03
openstackgerritMarcin Juszkiewicz proposed openstack/kolla master: nova-libvirt: handle ppc64le  https://review.openstack.org/43481011:03
openstackgerritMarcin Juszkiewicz proposed openstack/kolla master: ceph: move ceph-fuse package to cephfs-fuse image  https://review.openstack.org/44063411:03
openstackgerritMarcin Juszkiewicz proposed openstack/kolla master: kubernetes: disable for architectures other than x86-64  https://review.openstack.org/44563811:03
openstackgerritMarcin Juszkiewicz proposed openstack/kolla master: nova-compute: handle rtslib(-fb) for debian-binary builds too  https://review.openstack.org/43594111:03
openstackgerritMarcin Juszkiewicz proposed openstack/kolla master: debian: move to stretch  https://review.openstack.org/43445311:03
*** jistr is now known as jistr|afk11:06
nea1Hi, has anyone tested qos on ocata with ovs? - I'm setting a rule but there is nothing happening the logs look fine "Port cb3d964a-0f5f-4787-bd08-b245090643b4 updated. Details: {u'profile': {}, u'network_qos_policy_id': u'1fc76301-e0cd-477e-a2c6-4b062fdfef88',"11:13
nea1but it doesn't seam to have any effect11:13
nea1iptables on the node don't contain any new entries either11:14
*** tovin07_ has quit IRC11:15
*** matrohon has quit IRC11:16
hrwI will send a patch which will make some of centos/rhel/oraclelinux images fail. and this will be GOOD11:17
*** tovin07 has quit IRC11:19
*** manheim has joined #openstack-kolla11:21
*** athomas has joined #openstack-kolla11:21
*** satyar has joined #openstack-kolla11:23
nea1hrw: that actually doesn't sound GOOD11:23
hrwnea1: no, it will be good11:23
Serlexlol?11:24
hrwnea1: when you tell "yum install pkg1 pkg2" it can say "there is no pkg1" and install just pkg2 and be happy.11:24
hrwthen your image is useless as it does not contain pkg1 which is crucial to make image usable11:24
hrwapt in debian world in such situation says "no pkg1, cannot continue"11:25
hrwmy patch will make yum say "no pkg1, cannot continue" as well == broken images do not build == GOOD11:26
hrwINFO:kolla.image.build.openstack-base:No package Percona-Server-shared-56 available.11:26
hrwINFO:kolla.image.build.openstack-base:Error: Not tolerating missing names on install, stopping.11:26
hrwINFO:kolla.image.build.openstack-base:11:27
hrwINFO:kolla.image.build.openstack-base:Removing intermediate container d5e506330b0811:27
hrwERROR:kolla.image.build.openstack-base:Error'd with the following message11:27
nea1ok ^^11:27
nea1back to my qos test, it seams like in the netns there are iptable rules to mark the packages, but tc doesn't contain anything so the throttle never happens11:28
jaosoriormandre: thanks for updating the patch11:31
openstackgerritMarcin Juszkiewicz proposed openstack/kolla master: make yum fail on missing packages  https://review.openstack.org/44908611:32
hrwcan you guys take a look at that patch and plus it?11:33
nea1hm... no but iptables don't change if add a rule so it seams like the marks set there are not for qos11:33
*** masber has joined #openstack-kolla11:33
hrwsdake: https://review.openstack.org/44908611:35
* hrw off11:35
*** manjeets has quit IRC11:36
*** athomas has quit IRC11:43
*** vi01et03_ has joined #openstack-kolla11:50
*** vi01et03_ has quit IRC11:53
masberhi, I am having lots of issues making br-ex to work after a clean installation of openstack using kolla-ansible 4.0.0. I have been reading for a while and I think my problem is that openvswitch is not connecting br-ex with the openstack router gateway. I know troubleshooting these things are quite painful and takes a lot of time, but I really ran out of ideas and don't know what to do next. These are some of the details http://paste.openstack.org/s11:55
masberhow/603896/11:55
masberhttp://paste.openstack.org/show/603896/11:55
*** athomas has joined #openstack-kolla12:00
*** shardy is now known as shardy_lunch12:08
*** srwilkers has joined #openstack-kolla12:22
*** zhurong has quit IRC12:22
*** jistr|afk is now known as jistr12:27
*** matrohon has joined #openstack-kolla12:38
*** qiliang27 has quit IRC12:41
*** mkoderer has quit IRC12:47
sdakezhenguo any luck?12:48
sdakezhenguo disregard plz12:49
mnasero/12:49
mnasermorning12:49
sdakesup folks12:49
srwilkersmorning sdake12:49
sdakesup srwilkers12:49
sbezverk_good morning12:49
mnaserwe dont have a broken gate12:49
mnaserso thats a thing12:49
sbezverk_Jeffrey4l: ping12:49
mnaserdoes recheck automatically rebase12:50
mnaseri think this discussion was had yesterday12:50
*** athomas has quit IRC12:50
sdakemnaser yes12:51
sdakemnaser it doesn't rebsae the gerrit commit in gerrit12:51
sdakemnaser it does rebase on master before running the gate with zuul cloner12:52
*** schwicht has joined #openstack-kolla12:52
srwilkersnot much sdake, just enjoying the quiet office12:52
* portdirect throws things at srwilkers12:53
mnaserim going to recheck the stuff that is blocked by gate and got a +2 already12:53
*** dave-mccowan has joined #openstack-kolla12:54
sdakemnaser here is evidence: https://review.openstack.org/#/c/448043/12:55
sdakemnaser this patch was jenkins -1 on the check jobs prior to the recheck I did12:56
sdakemnaser no rebase involved12:56
*** jascott1 has joined #openstack-kolla12:56
sbezverk_sdake: srwilkers: appreciate your review of this one https://review.openstack.org/441550 !!! do not workflow it I need kfox1111 ok too.12:56
openstackgerritMarcin Juszkiewicz proposed openstack/kolla master: make yum fail on missing packages  https://review.openstack.org/44908612:56
hrwbug opened and mentioned in commit12:57
sdakesbezverk_ notice there is a gate failure12:57
sdakesbezverk_ any thoughts on that?12:57
*** shardy_lunch is now known as shardy12:57
hrwmnaser: great work on gate!12:57
sdakeoh right - this patch breaks subuntu12:57
sdakeon 2.0.2 it appears only?12:57
sbezverk_sdake: which one?12:58
sdakesbezverk_ the one you linked12:58
mnaserthank you hrw12:58
sdakehttps://review.openstack.org/#/c/441550/12:58
sbezverk_sdake: that job has NO any change, so it is not related12:58
sbezverk_sdake: if you check the log, it got stuck on ceph cluster thing, and ceph is not my thing ;)12:59
sdakesbezverk_ i did look at the log13:00
*** mnasiadka has quit IRC13:00
sdakei dontsee stuck on ceph - still learning how to understand these gates output13:00
*** jascott1 has quit IRC13:00
hrwmnaser: now debuging why jenkins gave -1 took me minute13:01
mnaserhrw the per-image build folder is so nice right?13:01
hrwmnaser: yes, it is13:01
mnaseri dont know whoever implemented it but much love13:01
mnaserall i did was enable it13:01
hrwmnaser: once I found it I fall in love13:01
mnaserno more 5 mb files opening in browser :>13:02
sbezverk_sdake: http://logs.openstack.org/50/441550/119/check/gate-kolla-kubernetes-deploy-ubuntu-binary-2-ceph-nv/4cb4600/logs/openstack/kolla-ceph-osd0.txt13:02
mnaserwe can thank harlowja for adding log_dir which makes gates clean today - https://bugs.launchpad.net/kolla/+bug/158647813:03
openstackLaunchpad bug 1586478 in kolla "Tech-debt: use image specific loggers" [Wishlist,Fix released] - Assigned to Joshua Harlow (harlowja)13:03
sdakesbezverk_ last night we had a question as to how to export the keystone-admin endpoint and keystone-internal endpoint - we have been through this before (me and you) and I couldn't recall what advice to give to duounghq - moment on logs:13:03
sdakehttp://eavesdrop.openstack.org/irclogs/%23openstack-kolla/%23openstack-kolla.2017-03-23.log.html#t2017-03-23T04:51:4613:04
openstackgerritMarcin Juszkiewicz proposed openstack/kolla master: openstack-base/kolla-toolbox: use mariadb-devel for CentOS and !x86-64  https://review.openstack.org/44483213:04
openstackgerritMarcin Juszkiewicz proposed openstack/kolla master: ceph: move ceph-fuse package to cephfs-fuse image  https://review.openstack.org/44063413:04
openstackgerritMarcin Juszkiewicz proposed openstack/kolla master: nova-compute: handle rtslib(-fb) for debian-binary builds too  https://review.openstack.org/43594113:04
sbezverk_sdake: tools/build_local_admin_keystonerc.sh13:05
hrwmnaser: I still open that 5MB file to find error but do not have to dig in it13:05
mnaserhrw it shouldnt have any logs at all13:05
mnaserconsole.log should be pretty much empty13:06
sdakesbezverk_ yes i know that command builds the adminrc13:06
mnaserunless there's a traceback or something13:06
sbezverk_sdake: for external consumtion, you need to run it with "ext" parameter13:06
hrwmnaser: ah, right - its 113K now13:06
sdakesbezverk_ read the log for our diag13:06
mnaser:D13:06
sdakedialog13:06
hrwmnaser: fast network connection and I did not noticed as new what I want to look for13:06
hrwmnaser: magic phrase usually is "Expected image"13:07
mnaseryeah exactly13:07
mnaseri should probably make a little write up about that at some point13:07
sdakesbezverk_ looking at that code, that sounds fine13:07
*** rwallner has joined #openstack-kolla13:07
sdakesbezverk_ the problem is admin privs are often needed to do many operations13:08
sdakesbezverk_ and when we went through debug (adding my cloud.yaml snippets) there was no longer an external ip for keystone13:08
*** rwallner has quit IRC13:08
sdakehttps://github.com/openstack/kolla-kubernetes/blob/master/tools/build_local_admin_keystonerc.sh#L1013:08
sbezverk_sdake: please ifyou see someone tries to resolve local problem by modifying common_workflow_config, stop them right away! they do something wrong13:08
*** rwallner has joined #openstack-kolla13:09
hrw3 more jenkins jobs and my patchset should be V:+1 :d13:09
sdakesbezverk_ well people gonna do what people gonna do when there are no docs ;)13:09
*** zhubingbing_ has joined #openstack-kolla13:09
sdakesbezverk_ its the only way to experience the code and understand the problems13:10
sbezverk_sdake: well, then do not complain ;)13:10
sdakesbezverk_ I wasn't ocmplaining13:10
hrwfound new bug with that yum change ;D13:10
sdakesbezverk_ I was asking a question - what cloud.yaml snippets are needed to get external keystone to work properly13:10
sbezverk_sdake: I was not refering to you, but to duonghq13:10
sdakesbezverk_ i understand13:10
sbezverk_just two lines are needed.13:11
sdakesbezverk_ the defaults do not work correctly for both the admin  (35357) and non-admin case (port 5000)13:11
*** haplo37 has quit IRC13:11
hrwironic-conductor was broken for centos/source builds and no one noticed13:12
sdakehrw you noticed :)13:12
hrwsdake: jenkins did13:13
sdakehrw openstack doesn't use jenkins13:13
sdakehrw openstack instead uses zuul13:13
sdakejenkins didn't scale to the 2kjobs/hour that openstack requires13:13
hrwsdake: whatever. mails are signed "Jenkins" :D13:14
sbezverk_sdake: here is my cloud.yaml and I highlighted what you need http://paste.openstack.org/show/603915/13:14
openstackgerritMarcin Juszkiewicz proposed openstack/kolla master: ironic-conductor: add missing comma for centos/source build  https://review.openstack.org/44913513:14
sdakesbezverk_ note i am reviewing your patch but struggling asi  justwoke up - brain booting13:15
sdakesbezverk_ there is this vague comment here on line 42: https://review.openstack.org/#/c/447356/5/doc/source/deployment-guide.rst13:16
openstackgerritChristian Berendt proposed openstack/kolla master: Fix name of the mistral-dashboard horizon plugin  https://review.openstack.org/44913613:16
sdakesbezverk_ it says to prune cloud.yaml13:17
sdakesbezverk_ I spoke with kevin on irc about it, he said to use generate_cloud_yaml_default (iirc) to generate just the absolute changes from the defaults that are needed13:17
sdakesbezverk_ i am struggling with this activity13:17
hrwso if someone has spare 2 minutes then please look at https://review.openstack.org/449135 one13:17
sbezverk_sdake: first time I hear about generate_cloud_yaml_default13:19
sdake# TODO(sdake): This file is generated by13:20
sdake#    /opt/kolla-kubernetes/tools/build_example_yaml.py13:20
sdake#    https://etherpad.openstack.org/p/kolla-cloudyaml13:20
sdake# The example should be pruned to just the changes needed to run, rather13:20
sdake# than force all defaults to their default value. Its hard to see what13:20
sdake# actually changed this way.13:20
sdakebuidl_exaple_yaml.py rather13:20
*** haplo37 has joined #openstack-kolla13:20
sdakehow does one remove tainting from the master node?13:21
sdakezhubingbing_ you awake?13:21
zhubingbing_hear13:21
zhubingbing_sbezverk_13:22
mnaserwow good catch hrw13:22
zhubingbing_we should change something in gate  ;)13:22
mnaserim trying to see how we can avoid having that happen again13:22
sbezverk_sdake: whatever you do, please make sure you leave build_example_yaml.py as is13:23
sdakesbezverk_ i wasn't going to modify it13:23
sdakesbezverk_ i was going to include it in th edocumentation hoepfully13:23
sdakesbezverk_ what i'd really like is to understand how to generate a cloud.yaml file without having a phd in kolla-kubernetes13:23
sbezverk_sdake: ah ok, because for my case it does, is making perfect sense13:24
sdakesbezverk_ huh?13:24
sbezverk_zhubingbing_: hey, you pinged me?13:24
sdakesbezverk_ we are all attempting to sort out how to deploy kolla-kubernetes outside a dev env and outside the gate13:25
sdakesbezverk_ all the pings re the same thing13:25
sbezverk_sdake: sorry, I mean build_example_yaml.py is making perfect sense13:25
sdakesbezverk_ ok well it may make perfect sense in your head, but if its not documented it doesn't make sense at all13:25
sdakecan you help me with wording for that TODO above13:25
sbezverk_sdake: cloud.yaml I pasted you is from my local kube cluster13:25
sbezverk_it has been tested inside out13:26
sbezverk_can you use it as a base?13:26
sdakesbezverk_ right - kfox1111 had indicated instead I should trim that to the bare minimum13:26
sdakehe siad to use build_example_yaml.py to do that13:26
sdaketrim that = trim cloud.yaml13:26
sdakesbezverk_ i could use your cloud.yaml as a base, although it has a bunch of defaults in it13:27
sdakesbezverk_ kfox1111 's position was there is no point in defining  the defaults twice13:27
sdakein the documentation13:27
sdakewhich sounds reasonable13:27
sdakehow to do that - I dont know13:27
sdakeI need a trimmed cloud.yaml13:27
sbezverk_sdake: ok I understand what you are saying, I do not understand a reason, because that cloud.yaml is primary configuration tool for a person who deploy kolla-kube13:27
sdakeand a technique to do so13:27
sbezverk_if you trim default options then that person becomes unaware what can be tweaked13:28
sdakesbezverk_ the reason given in the review was that the operator should be able to see what the minimum stuf fis that is needed to get going13:28
sdakesbezverk_ versus the "here are the 100 options needed"13:28
sdakesbezverk_ i can document both types of workflow, however, i see merit in understadning just the deltas13:29
openstackgerritMarcin Juszkiewicz proposed openstack/kolla master: ironic-conductor: add missing comma for centos/source build  https://review.openstack.org/44913513:29
hrwbug filled, commmit updated13:29
sbezverk_sdake: I come from a different world, I need to see all my options and I change only what I need13:29
sdakesbezverk_ right - thats cool - i can document that too13:29
sdakesbezverk_ i personally would like to understand the deltas and I think other operators do too13:29
*** Serlex has quit IRC13:29
sdakesbezverk_ the problem is we have hundreds of options and none of them are documented13:30
sdakesbezverk_ so changing just what you need involves looking deeply into helm charts13:30
sdakesbezverk_ and understanding what that means13:30
sdakesbezverk_ this is a QSG for deploy not a PHD dissertation on deployment :)13:30
sbezverk_sdake: what do you need from me then?13:30
sdakewe can get to documenting all the options later13:30
sdakesbezverk_ a trimmed down delta of just the minimal otpions needing configuration in cloud.yaml13:31
sdakesbezverk_ as in a minimal cloud.yaml13:31
sdakesbezverk_ or a technique to generate one13:31
sdakeor and a techniue to generate one13:32
*** mbruzek has joined #openstack-kolla13:32
*** mbruzek has joined #openstack-kolla13:32
sdakei mean I could put yours in the repo as the cononical example, and then diff against whatever is generated in cloud yaml creator or whatever its called13:32
sdakeand then document the diff13:32
sdakewoudl that be the correct approach?13:32
sdakeI dont need you to do the work, I need to know how o do it :)13:33
sbezverk_sdake: since I have never thought about it, so I do not know how..  the thing is, that cloud.yaml is very highly customized. even things like image tags, which would not be the case if we did not default to 2.0.2 images etc13:34
sdakeok - well action to update the defaults from 2.0.2 to somethign more recent13:35
rwellumwe need to start with a solid and clear description of each important option and how they inter-relate imo. Some of them are not obvious to me.. 'external_vip' for example.13:35
sbezverk_sdake: I am afraid when you run diff you will get 80%+ differences13:35
sdakethere is zero reason the defaults should be 3 cycles bck13:35
sdakerwellum ++++++13:35
sdakesbezverk_ rwellum hit it on the head -some options don't matter13:36
sdakesbezverk_ some do13:36
sdakesbezverk_ i'd liek to focus on the ones that do13:36
sbezverk_rwellum: some names used were due to the baggage we got from kolla-ansible13:36
sdakesbezverk_ thats fine - nobody cares about why names were used13:37
sdakewe care about what the names mean13:37
sbezverk_rwellum: I would call it differently, but it is how it/was in kolla ansible13:37
sdakenames being variables in this case13:37
sdakefor example in kolla-ansible docs here:13:38
sdakehttps://docs.openstack.org/developer/kolla-ansible/quickstart.html13:38
rwellumsbezverk_: ack13:38
sdakehttp://paste.openstack.org/show/603926/13:39
sdakeneed that same sort of thing for kolla-kubernetes13:39
sdakewhat options must one configure to get from defaults to a working system13:39
sdakefwiw the defaults should yield a working system :)13:40
sdakethey dont today - which is fine - so lets document how what gets a working system13:40
sbezverk_sdake: absolutely NOT13:40
*** jtriley has joined #openstack-kolla13:40
sdakesbezverk_ i said 5 different things there which one do you object to13:40
sbezverk_sdake: common, how default can get you a working system, if you need your personal IP to attach to>?!?!?!?13:40
sdakesbezverk_ right -t here is always some minimal level of configuration needed13:41
sdakewhat is that minimal level of configuraiton13:41
sdakesee past eabove13:41
sdakethis is an exmaple from the kolla-ansible quickstart13:41
*** skramaja has quit IRC13:41
sdakein kolla-ansible ther eare like 5 config value sthat need to be configured to obtain a functional system13:41
sbezverk_sdake: setting you mentioned are not "deployment" setting, at best for dev env.13:42
sdakesbezverk_ listen dude Im super proud of th ework the community has done - I merely want to document it so others can enjoy in my proudness :)13:42
sdakesbezverk_ ok - well then best for dev env - gotta start somewhere :)13:43
sdakesbezverk_ atm we are blocked on debating whether we should even document cloud.yaml or not13:43
sdakeright now cloud.yaml is a black box13:43
sbezverk_sdake: since you plan to document it, I do not see what stopping you just do it :) I am not twisting your arms or something13:43
*** krtaylor has joined #openstack-kolla13:44
sdakesbezverk_ the faster i learn how cloud.yaml works and the faster rwellum learn's how cloud yaml works and he faster zhubingbing_ learns how cloud.yaml works, the faster we learn together13:44
sdakesbezverk_ I dont want to document the whole thing13:44
sdakesbezverk_ just the basics13:44
sdakesbezverk_ I don't know what the basics are13:44
zhubingbing_ok13:44
sdakesbezverk_ hence I need some help here13:44
sdakesbezverk_ you mentioned the ip to attach to13:44
sdakethat is a basic13:44
sdakeso how about something like this:13:45
sdakestep 1. generate cloud yaml13:45
*** Margin has joined #openstack-kolla13:45
sdakestep 2. configure IP (how)13:45
sdakestep 3. configure keystone for external access (how)13:45
*** manheim has quit IRC13:45
sdakestep 4. set image tags since the defaults are borked (how)13:45
sdakei need to know how to do these things13:45
sbezverk_sdake: and who is going to take care of how part?13:45
sdakeif you explain it to me, i will writ ethe docs13:45
zhubingbing_yeah13:46
sdakesbezverk_ i was hoping you would tell me13:46
sdakesbezverk_ or someone would tellm e that knows13:46
sdakesbezverk_ this is the #1 blocker in my mind for producinga deployment guide13:46
sdakeor whatever doc we are producing here :)13:46
*** manheim has joined #openstack-kolla13:46
sbezverk_sdake: I am sorry but I do not have cycles to do it. You know it is not my main job and I have other things on going.. I can13:46
sbezverk_helm answering "some" questions13:47
sbezverk_s/helm/help/13:47
rwellumAs a minimum,  'external_vip': your external ip address, 'ext_interface_name', 'ext_bridge_name' and 'tunnel_interface' - is what I had to change13:47
rwellumFYI13:47
sdakerwellum thanks - this is the kind of info I am looking for sbezverk_13:47
sdakerwellum did you manage to bring up kolla-kubernetes?13:47
sdakeso lets start with these 4 variables13:48
rwellumsdake: I got stuck at this step ;)13:48
sdakesince rwellum had to chang eem13:48
sdakesbezverk_ ^^ everyone gets stuck at this step including me13:48
sdakewe know we need to change these 4 variables -  what should they be changed to?13:48
rwellumThere was a mistake in the file that someone pointed out in the review. But like sbezverk_ I got pulled off into other things - I'm ready to try again tho.13:48
sdakerwellum good to hear :)13:49
sdakerwellum I was travelling for 2 weeks so feel your pain :(13:49
rwellumAlso inc0 was trying the last I saw I was sitting back and watching the experts figure it out :)13:49
sdakeok - well waiting for someone else to solve ap roblem doesn't solve a problem ;)13:49
rwellumtruism13:50
sdakewe need to solve it together13:50
sbezverk_external_vip: ip address assigned to your host and kubernetes api server listen on13:50
sdakesbezverk_ ++ :)13:50
sbezverk_sdake: funny thing, I do not know how many times I have already mentioned it, something tells me it will not be the last one13:50
*** fguillot has joined #openstack-kolla13:51
sdakesbezverk_ it is being written down13:51
sdakesbezverk_ and published on docs.oo13:51
sbezverk_sdake: you wrote it down numerous time, and here we are again13:51
sdakei am writing your comments in this review as you type them13:51
sdakehttps://review.openstack.org/#/c/447356/6/doc/source/deployment-guide.rst13:51
sdakethat guide will be published on docs.oo13:52
rwellumsbezverk_: you've told me numerous times too - sorry - some I wrote down and some I didn't mainly due to lack of knowledge13:52
sdakeso this question will be asked once and answered13:52
sbezverk_ext_interface_name: name of interface or bridge which will be used by neutron's provider interface13:53
hrwbye13:53
sdakelater hrw13:53
sbezverk_ext_bridge_name: name of bridge you want neutron to use as external bridge, by default it shold be br-ex, but you can change it by using this option13:54
sbezverk_tunnel_interface: interface name used for connectivity between nodes in kubernetes cluster, in most of cases it matches the name you kubernetes host management interface13:55
sbezverk_well, here is the problem, if you do not look in the code for these things, then how you plan to learn the code and then develop??13:56
sbezverk_hopefully it will unblock you ;)13:57
sdakenote comments in https://review.openstack.org/#/c/447356/6 on line 2413:57
sdakesbezverk_ there are hundreds of variables - looking at code for all of them is not scalable from a developer POV - from a deployer/operator POV, they often don't care about the code or hae time to decode it13:58
sdakesbezverk_ can you think of other variables we need to set?13:58
sdakesbezverk_ we identified 4 that needed to be changed always13:58
sbezverk_sdake: storage options13:59
*** schwicht has quit IRC13:59
sdakesbezverk_ anything else?14:00
*** lucker has joined #openstack-kolla14:01
sbezverk_sdake: I think the rest is not extremely essential..14:02
sdakegood to hear :)14:02
sdakerwellum what tz are you in?14:03
rwellumI'm in RTP sdake14:04
rwellumEST14:04
sdakerwellum roger est then14:04
*** Serlex has joined #openstack-kolla14:04
sdakerwellum i am MST14:04
rwellumcool14:05
rwellumsbezverk_: this is really useful - thanks14:05
*** rabel_b1 has joined #openstack-kolla14:05
sdakerwellum up for some collaboration on producing a deploy today via webex?14:05
sdakerwellum i'm struggling to write this document - and could use a second set of eyeballs14:06
sdakezhubingbing_ you will probablyb easleep by the time I have open slots on my calendar14:06
sdakezhubingbing_ if you have any heavy wizardry experience to add, can you add it to the review?14:07
*** rabel has quit IRC14:08
*** lucasxu has joined #openstack-kolla14:08
*** Margin has quit IRC14:09
sdakesbezverk_ re the storage config options14:09
sdakesbezverk_ mind sketching those out for us?14:09
*** caoyuan has joined #openstack-kolla14:10
zhubingbing_yeah14:13
zhubingbing_please add me to the revierwer14:13
caoyuanhi14:14
sbezverk_sdake: sorry very low on available time, since I am on that training program. you ask specific questions, I will try to answer14:15
*** yangyape_ has joined #openstack-kolla14:15
yangyape_caoyuan: ping14:15
sdakesbezverk_ oh sitll in training?14:15
sdakesbezverk_ sounds good re answering specific Q&A14:15
yangyape_caoyuan: hi14:16
sbezverk_sdake: until first week of june, when we need to preent our project14:16
sdakesummit is 5 weeks out14:19
spsuryasdake:  yep14:19
spsuryasdake:  many of the contributors must be facing US VISA approval issue :(14:21
sdakespsurya possible14:22
sdakespsurya thankfully next summit is in canada which is more lenient14:22
*** goldyfruit has joined #openstack-kolla14:23
Serlexnot Sydney?14:23
sdakesorry sydey then canda14:23
sdake(vancouver)14:23
Serlexthat would be nice14:23
sdakenot sure what sydney's visa process is like - US typically has visa-free entry to most countries14:23
sdakethe only place I have ever applied for a visa before is china14:23
spsuryasdake: yes14:24
spsuryaits Sydney14:24
sdakeya summit after is vancouver14:24
sdakefoundation indicated via superuser they were having all summits in 2018 outside the US14:25
*** fguillot has quit IRC14:25
sdakesuperuser article thatis14:25
Serlexbummer14:26
sdakeSerlex bummre why?14:27
*** shashank_t_ has joined #openstack-kolla14:28
spsuryasdake: roger14:28
SerlexI only say that because how popular Openstack is in US14:30
rwellumsdake: I am fine to webex together whenever you need, except 2-4pm EST.14:32
openstackgerritOpenStack Proposal Bot proposed openstack/kolla master: Updated from global requirements  https://review.openstack.org/44677114:33
sdakerwellum i have to get the kids to school from now for about 1 hr, and then have two hours of meetings (which ends at 2pm EST if I did the TZ math correctly)14:33
sdakeif 4pm EST onwards ok for you?14:34
rwellumsdake yup14:34
sdakerwellum i'm farily convinced if we can get some reasonable docs written on how this stuff works - we will be able to make forward progress as a community14:35
sdakerwellum cool - ill ping you as soon as my torture sessions end for the day  :)14:35
sdakerwellum actually since yours are after mine, perhaps you should ping me14:35
*** shashank_t_ has quit IRC14:36
rwellumsdake: agree and sounds good I will.14:36
*** shashank_t_ has joined #openstack-kolla14:36
rwellumYour environment or mine?14:37
sdakerwellum both ? :)14:37
sdakerwellum whatever works best for you - we can use yours14:37
sdakerwellum two of my machines in my home lab seem to have gone out to lunch14:37
*** dciabrin has quit IRC14:38
sdakeso that may add more complication to getting us going as I only have one bare metal node to work with14:38
*** dciabrin has joined #openstack-kolla14:38
rwellumYes same in that I have to use VM's for the most part.14:38
dmsimardSerlex: political instability/uncertainty is my guess14:39
*** yangyape_ has quit IRC14:45
*** yangyapeng has joined #openstack-kolla14:47
*** fooliouno has joined #openstack-kolla14:48
*** shashank_t_ has quit IRC14:49
*** athomas has joined #openstack-kolla14:49
*** schwicht has joined #openstack-kolla14:51
*** yangyapeng has quit IRC14:52
*** manheim has quit IRC14:54
mnaserhttp://grafana.openstack.org/dashboard/db/kolla-failure-rate?from=1490194436373&to=149028083637314:55
mnaserweee lets try to get that as low as possible14:55
mnaseri still wonder why rhel-based is around 10% higher on failures usually14:55
*** caoyuan has quit IRC14:58
spsuryanight all14:59
sdakemnaser how did you set that up?14:59
mnasersdake https://review.openstack.org/#/c/446697/15:01
mnaseranyone can add grafana graphs15:01
*** adrian_otto has joined #openstack-kolla15:03
*** zhubingbing_ has quit IRC15:04
*** sayantani01 has joined #openstack-kolla15:06
*** zhubingbing_ has joined #openstack-kolla15:13
*** matrohon has quit IRC15:14
*** matrohon has joined #openstack-kolla15:15
*** new2Kolla has joined #openstack-kolla15:15
openstackgerritMerged openstack/kolla master: Speed up builds by increasing threads for builds  https://review.openstack.org/44670115:16
sdakehey fooliouno15:17
kfox1111morning.15:17
rwellumSource build failure rates at 70%? Was that just a blip - seems crazy high? Be cool if the graph had a weekly, monthly view.... mnaser ?15:17
sdakefooliouno we had some discussion this morning around the docs15:17
mnaserrwellum you can click in the top right and select timelines15:17
sdakerwellum click "zoom out"15:17
mnaserpast N hours ,etc15:17
sdakerwellum ya our gate was wedged15:18
mnaserand yeah, gate was blocked till this was done - https://review.openstack.org/#/c/448666/15:18
sdakeironic upstream was broken15:18
sdakeand possibly something else :)15:18
mnaserim mostly happy about the log-file-per-image15:18
mnaserfinding image build problems is so much easier now15:18
sdakerather biforst upstream15:18
*** jtriley has quit IRC15:18
kfox1111ah. nice to see openstack adopting grafana15:18
mnaserkfox1111 you can add kube graphs in there too!15:19
kfox1111nice. :)15:19
mnaseri just dont know much about how the ci works there to muck around it but yeah it's a useful thing to have15:19
mnaserhttps://git.openstack.org/cgit/openstack-infra/project-config/tree/grafana/kolla.yaml15:19
*** shashank_t_ has joined #openstack-kolla15:19
mnaserhopefully once its at reasonable #s we can make kolla-ansible deploy gates vote15:19
kfox1111that link mentions grafyaml and grafana in project-config.15:20
mnaseryeah, you can just propose a change review just like any other openstack change15:21
rwellumAh that's what zoom-out means. :)15:21
mnaseralso guys cool idea15:21
kfox1111looks pretty much like a yaml dump of the grafana dashboard.15:21
mnaserhttps://github.com/openstack-dev/devstack/tree/master/files/rpms15:21
mnaserwhat if we look into adopting that15:21
mnaserhttps://github.com/openstack-dev/devstack/tree/master/files/debs15:21
mnaserthat way we dont have to sit and maintain which packages we need15:21
mnaserits all in one centralized spot..15:21
sdakemnaser i dont follow15:22
mnaserfor example, the other day source deploys were broken because haproxy become a requirement15:22
openstackgerritMerged openstack/kolla master: Remove py34 settings from gating  https://review.openstack.org/44724715:22
mnaserwe didnt have it installed so all deploys failed.15:22
mnasersay if we relied on something like that, it would have been added here, and we'd have picked up the change - https://github.com/openstack-dev/devstack/blob/master/files/debs/neutron#L515:22
mnaserand the effort of maintaining which project needs which packages is simplified and unified across devstack and kolla15:23
sdakeare you proposing to reimplement EPEL/RDO/UCA? :)15:23
mnasersource builds dont install dependencies15:23
mnaserwe install all these dependencies manually15:23
sdakeyes, and they come from the RDO and UCA repos15:23
mnaserwhat im saying is15:24
sbezverk_kfox1111: good  morning, have you seen my reply about nova cells?15:24
mnaserinstead of us maintaining a list of packages and devstack maintaining (effectively the same list)15:24
kfox1111sbezverk_: no. in the scroll back?15:24
mnaserhttps://github.com/openstack-dev/devstack/blob/master/files/debs/neutron vs https://github.com/openstack/kolla/blob/1d9b41f3d8ef05ea1b9516a6e853eb7f84c67e18/docker/neutron/neutron-base/Dockerfile.j2#L66-L7915:24
*** manheim has joined #openstack-kolla15:25
sdakeinstead of maintaing separately maintain where?15:25
kfox1111sbezverk_: a huge amount of scroll back this morning. could you summerize?15:25
openstackgerritOpenStack Proposal Bot proposed openstack/kolla master: Updated from global requirements  https://review.openstack.org/44677115:26
mnasersdake either use what devstack uses, or propose to share it somehow.15:26
EmilienMcan someone approve https://review.openstack.org/#/c/448043/ ? it has 3 °215:26
EmilienM+215:26
sdakeEmilienM we are waiting on the gate to go green which it should15:27
*** jtriley has joined #openstack-kolla15:28
sdakeEmilienM usually this can be achieved thorugh rechecks - although the implementation may be off slightly15:28
sdakelooking at the problem now15:28
mnasersdake its all green, except it looks like a non voting deploy job failed15:28
mnaserwhich should be ok15:28
mnaserconsidering it seems like mariadb bootstrap failed15:28
EmilienMsdake: it's green now.15:28
EmilienM?15:28
sdakejust because its non-voting doesn't mean we should have a regression in the gate jobs15:29
mnaserbut gate doesnt even run deploy jobs15:29
sdakeEmilienM this gat ejob fails: http://logs.openstack.org/43/448043/1/check/gate-kolla-dsvm-deploy-oraclelinux-binary-centos-7-nv/a74a093/15:29
EmilienMnot sure it's related to my patch15:29
mnaserthats a check job, and its non voting because its unreliable, and it seems pretty clear that the cause is not that15:29
EmilienMmy patch is about RDO repos15:29
sdakeEmilienM agreed that is why we recheck15:29
sdakeEmilienM oraclelinux uses RDO repos15:30
EmilienMwell, I'm switching http to https15:30
sbezverk_kfox1111: it is in review reply15:30
EmilienMnot sure it's related but ok...15:30
sbezverk_should be in your mail box15:30
kfox1111sbezverk_: ah ok. sec15:30
sdakeEmilienM yup - our gate is a little flakey15:30
sdakeEmilienM we sitll dont want to have a regression so we treat the gate as voting even though it doesn't vote15:30
*** hrw has quit IRC15:30
sdakeEmilienM we are moving the gating to voting as soon as feasible15:31
EmilienMwell, I work on RDO every day and I can tell that using https instead of http will help you to be more stable15:31
EmilienMwe saw some cases where redirect doesn't work15:31
kfox1111sbezverk_: ah. ok. that explains it, and makes me feel much better. :)15:31
sdakeEmilienM all good - i am looking at the specific failure now hang tight15:31
kfox1111I guess simple setup does that.15:31
sdakemnaser changed how the gate logs re ecorded so I need to learn that now :)15:31
mnasersdake only for image builds15:32
kfox1111thanks for getting verification.15:32
mnaserthis was a deploy15:32
mnaserso everything is still in html15:32
mnaserconsole.html15:32
mnaseri checked it and mariadb failed to bootstrap15:32
mnaserhttp://logs.openstack.org/43/448043/1/check/gate-kolla-dsvm-deploy-oraclelinux-binary-centos-7-nv/a74a093/logs/kolla/mariadb/mariadb.txt.gz15:33
mnaserERROR 2003 (HY000): Can't connect to MySQL server on '10.31.184.114' (111 "Connection refused")15:33
kfox1111sbezverk_: I think for the iptables stuff, since your using ovs now, all you may need to do is add some iptables INPUT rules for allowing traffic from your tenant bridge.15:34
mnaseractually that change has zero impact on that job15:34
mnaserbecause the deploy job downloads images instead of building them15:35
*** hrw has joined #openstack-kolla15:36
sbezverk_kfox1111: it seems it is not required as traffic goes in/out fine..15:36
*** jascott1 has joined #openstack-kolla15:37
sbezverk_adding complexity when it is not absolutely requiered is not great approach15:37
*** manheim has quit IRC15:38
kfox1111sbezverk_: could be. I think I rewrote the iptables rules to mostly only apply to the external interface.15:38
*** schwicht has quit IRC15:39
sbezverk_kfox1111: it would have been broken if it were really required15:40
sdakemnaser the job is not non-voting because it is unreliable15:40
kfox1111true15:40
mnasersdake im aware, but if we just recheck it till it goes green. we're not really solving the issue15:40
mnaserand we're using up resources for no reason15:40
kfox1111sbezverk_: initramf_url: is the f a typo?15:40
mnaseri'm all for making jobs vote again, but by looking into the issues and resolving them15:40
kfox1111or a missing 's'?15:41
sdakemnaser wfm - jeffrey4l and inc0 are leading the gating work15:41
sdake(for kolla-ansible)15:41
sdakeegonzales is helping as well15:41
sdakei'm not really involved15:41
sdakejust  following the process we have in place15:41
sbezverk_kfox1111: yep, itshould be initramfs15:41
kfox1111sbezverk_: k. I'll mark it.15:42
sbezverk_kfox1111: but it is missspelled consistently ;)15:44
jaosoriormandre: will you file the LP bug that was requested here https://review.openstack.org/#/c/446911/ ?15:45
kfox1111sbezverk_: yeah. :)15:45
*** new2kolla_ has joined #openstack-kolla15:47
*** new2Kolla has quit IRC15:47
*** shasha_t_ has joined #openstack-kolla15:48
*** shashank_t_ has quit IRC15:52
*** shashank_t_ has joined #openstack-kolla15:54
*** lamt has joined #openstack-kolla15:56
*** david-lyle_ is now known as david-lyle15:58
*** mliima has joined #openstack-kolla15:59
mliimahello!16:00
*** jtriley has quit IRC16:00
*** targon has quit IRC16:01
*** matrohon has quit IRC16:06
*** unicell has joined #openstack-kolla16:07
*** unicell1 has quit IRC16:09
*** jascott1_ has joined #openstack-kolla16:09
*** lucker has quit IRC16:09
*** jtriley has joined #openstack-kolla16:11
mandrejaosorior: do you mind filing that bug? sorry, I'm a bit short on time16:14
sbezverk_kfox1111: how do you suggest we proceed with that ps? I change initramfs and then we can merge it?16:14
jaosoriormandre: we all are. No biggie; I'll file it.16:15
jaosoriormandre: thanks again for updating the patch.16:15
openstackgerritMerged openstack/kolla-ansible master: Update the "virt_driver" to "virt_type"  https://review.openstack.org/44733616:15
*** manheim has joined #openstack-kolla16:16
openstackgerritMerged openstack/kolla-ansible master: Fix mongodb cluster using wrong network interface  https://review.openstack.org/44299916:19
openstackgerritMerged openstack/kolla-ansible master: Add an extra network node for HA  https://review.openstack.org/44849416:20
inc0good morning16:24
mandrejaosorior: I'm thinking we'll need a release note for your SSL patch16:25
jaosoriormandre: no biggie man, I'll add that once I add the LP bug.16:25
*** matrohon has joined #openstack-kolla16:26
mandrejaosorior: perfect, thanks16:26
*** jascott1_ has quit IRC16:30
*** lucasxu has quit IRC16:32
hrwre16:32
hrwfeel strange. all patches got +1 from CI... and one which did not failed because of already-present-but-somehow-missed bug16:33
*** mbruzek has quit IRC16:33
hrwhm... that --threads patch which got recently merged will be fun on ppc64le machines16:36
hrwah. tht's tests only16:36
sbezverk_inc0: good morning, qq can you change config gen ps to write config files in a different from /etc/kolla location?16:39
hrwcan someone give +2 to https://review.openstack.org/449135 so we can get https://review.openstack.org/449086 verified and then +2 and merged?16:41
*** rhallisey has quit IRC16:44
*** adrian_otto1 has joined #openstack-kolla16:46
inc0sbezverk_: check out my patchset - you have CONFIG_DIR=/etc/kolla in ansible-playbook call16:47
inc0you can change this to whatever you want16:47
egonzalezhrw, after fixing ironic build you will find another error and you know it ;)16:48
sbezverk_inc0: it is not exactly what I had in mind :)16:48
inc0then what do you have in mind?:)16:49
sbezverk_would be breat to be able to write configs in helm/microservice/{microservice-name}-configmap/data ;)16:49
*** adrian_otto has quit IRC16:49
hrwegonzalez: sure, but ironic-conductor bug keeps yum from entering. then I will have to deal with lot of centos failures but that's excepted anyway. and with yum fix being in I will have errors in proper places == less debugging == more work done16:49
inc0well, doable, I can make this happen...hell you can use my other patchset to write rendered configmap16:50
inc0and template that16:50
inc0but copying isn't big deal...templating is16:51
sbezverk_inc0: I do not give up the idea to use helm16:51
inc0neither do I16:51
*** rhallisey has joined #openstack-kolla16:51
inc0what I'm saying is, when we do it, we should do it correctly16:51
sbezverk_inc0: placing config files into that folder would allows us to do it very very easily16:51
inc0well it's really simple playbook16:52
inc0like this one https://review.openstack.org/#/c/447733/16:52
inc0I can change it to output either fully rendered configmap or just internal data16:53
inc0but I don't really like this approach tbh16:54
inc0if we're getting rid of ansible, let's get rid of it alltogethe16:54
inc0r16:54
inc0and that means moving templating of configmaps all the way to helm16:54
sbezverk_inc0: atm there is a limitation in helm :( it can access files only located inside of chart folder16:55
inc0sbezverk_: what I'm saying is, I'm -1 to adding files dynamically to tree16:56
inc0either we render them all in helm16:57
inc0(including templates)16:57
inc0or we add them to k8s externally16:57
inc0reason I don't want to change things in tree is git16:57
inc0and stuff like that16:57
inc0code tree should not be touched16:57
*** egonzalez has quit IRC16:58
openstackgerritMichal Jastrzebski (inc0) proposed openstack/kolla-kubernetes master: WIP Move config generation to kolla-k8s  https://review.openstack.org/44825817:00
*** new2kolla_ has quit IRC17:01
openstackgerritMichal Jastrzebski (inc0) proposed openstack/kolla-kubernetes master: WIP: Playbook to add configmaps and secrets  https://review.openstack.org/44773317:03
inc0sbezverk_: I addressed you comment17:03
*** sean-k-mooney2 has joined #openstack-kolla17:05
*** sean-k-mooney2 has quit IRC17:05
*** schwicht has joined #openstack-kolla17:06
*** unicell has quit IRC17:06
openstackgerritJuan Antonio Osorio Robles proposed openstack/kolla master: Prevent apache from listening on port 80  https://review.openstack.org/44903917:08
openstackgerritJuan Antonio Osorio Robles proposed openstack/kolla master: RHEL: Add mod_ssl for services running over httpd  https://review.openstack.org/44691117:08
openstackgerritJuan Antonio Osorio Robles proposed openstack/kolla master: Prevent apache from listening on port 80 for heat-base  https://review.openstack.org/44904017:08
jaosoriormandre: done ^^17:09
jaosoriormandre: added you as a co-author of the patch that adds mod_ssl17:09
*** pbandark has joined #openstack-kolla17:14
*** pcaruana has quit IRC17:14
openstackgerritMerged openstack/kolla master: Implement Keystone zero-downtime upgrade  https://review.openstack.org/42583317:17
sdakeinc0 the key is making defaults via helm config maps and making non-defaults via helm config maps as well17:25
sdakekfox1111 inc0 sbezverk_ you around to ddiscuss that now?17:25
*** manheim has quit IRC17:27
*** jmccarthy has quit IRC17:29
*** jaosorior has quit IRC17:30
*** tonanhngo has joined #openstack-kolla17:30
mnaserhrw got me worried that my threads would have broke stuff :D17:31
inc0sdake: I'd like to have single way to do it17:32
*** lucasxu has joined #openstack-kolla17:32
inc0if we do ansible, we do ansible17:32
inc0if we do helm, we go for it17:32
sdakeyou said you woudln't be married  to ansible...17:32
inc0I'm not married to ansible, I'll do it as quick fix17:32
inc0but what I don't want to do is some middle ground prothesis just to say "we do helm" even if we don't really17:33
sdakeright - so ansible eventuallyg et s ejected17:33
sdakeif it can't be done it can't be done17:33
inc0we need to move it to helm, that's true, but when we do it we do it properly17:33
hrwmnaser: that was before I noticed that it tests.17:33
sdakei think it can be done17:33
hrwmnaser: I was running kolla on 80 cpu system17:34
kfox1111sdake: yeah.17:34
inc0https://review.openstack.org/#/c/448258/ <- this took 2 days of work17:34
inc0that's why I'd rather go with ansible17:35
inc0now17:35
inc0we *might* go with ansible for 1.0, depends how we do17:35
inc0but moving it all to helm is priority17:35
hrwbye17:35
inc0good night hrw17:36
openstackgerritMerged openstack/kolla master: Updated from global requirements  https://review.openstack.org/44677117:36
kfox1111inc0: +1 for just copying from kolla-ansible to kolla-kubernetes for now, just to decouple us, prevent further breakages, and simplify the deployment. thats a lot of benifit for very little initial work.17:37
inc0exactly17:37
*** jtriley has quit IRC17:39
*** krtaylor has quit IRC17:42
*** jtriley has joined #openstack-kolla17:44
*** Serlex has quit IRC17:45
openstackgerritMerged openstack/kolla master: Switch trunk/cbs/buildlogs to use https  https://review.openstack.org/44804317:45
mnasergood and simple +W for any cores https://review.openstack.org/#/c/449135/17:47
openstackgerritSerguei Bezverkhi proposed openstack/kolla-kubernetes master: Virtualbmc and deployment of baremetal VM using ironic  https://review.openstack.org/44155017:48
*** deadnull has joined #openstack-kolla17:51
sbezverk_kfox1111: answered you latest comments17:51
*** adrian_otto1 has quit IRC17:52
sdakeinc0 i think we all agree ejecting kolla-ansible as a dep is mandatory - and the approach you took to do so is solid17:52
sdakeinc0 i think we may disagree that ansible should be a dependency at all - i'm not sure17:52
sdakeI don't think it is mandatory17:52
sdakeyou indicated for 1.0.0 it is mandatory17:53
*** rhallisey has quit IRC17:53
sdakei think this is open for debate if someone comes along and implements external config maps - because this removes most of the need for any type of ansible in the codebase17:53
sdakeinc0 what we really want to avoid as a team is a workflow using ansible17:53
sdakeinc0 if we were to do workflow with ansible we would be permanently locked into ansible as a dependency17:54
inc0sdake: workflow using ansible is still better than alternative we have now17:54
*** rhallisey has joined #openstack-kolla17:54
sdakeinc0 workflows shipping with 1.0.0 = cli = permanent thing17:54
sdakethat needs to be maintained and dicctates the permanent  architecture of the system17:54
kfox1111sbezverk_: looks good to me. :)17:54
sdakethat is why I asked you if you would be married to ansible17:55
sdakeyou said no17:55
sdakeand i hope you stick to that ;)17:55
openstackgerritMichal Jastrzebski (inc0) proposed openstack/kolla-kubernetes master: Move keystone configuration to kolla-k8s  https://review.openstack.org/44925417:55
*** mbruzek has joined #openstack-kolla17:56
*** mbruzek has quit IRC17:56
*** mbruzek has joined #openstack-kolla17:56
inc0sdake: I hope you already saw that I'm just trying to do what makes sense17:56
inc0if something else is better, I'll drop my approach happily17:56
*** adrian_otto has joined #openstack-kolla17:56
sdakeinc0 right - pragmatism is good, dictating a design of workflow in ansible is not something we want to do - we already had broad consensus on this point17:57
kfox1111inc0: I'd be happy to get the kolla-ansible templates -> kolla-kubernetes thign done asap. that should be pretty quick to do.17:57
kfox1111the other one, replacing kollakube is going to be a lot of work.17:57
inc0kfox1111: I just submitted another service;) 30min of work17:57
kfox1111realitive to the first one.17:57
kfox1111inc0: cool. :)17:57
inc0I don't believe it's going to be *that* much of work17:58
inc0I mean kollakube does 2 things really -create configmaps and create secrets17:58
kfox1111inc and theres some services that still need transition.17:58
*** jtriley has quit IRC17:58
inc0like what?17:58
sdakeI dont care about replacing kollakube i care about not being locked into ansible permanently17:58
kfox1111keepalived, half of ceph, and openvswitch-set-ip17:59
kfox1111sdake: +1.17:59
kfox1111kollakube isn't too bad. depending on genconfig's pretty painful.17:59
inc0approach I'm taking is - as long as files lands in /etc/kolla17:59
inc0I don't care who creates there17:59
kfox1111inc0: +1.17:59
inc0them*17:59
inc0so no, we won't be locked to ansible ever17:59
sdakeright - you have sent messages on irc that indicate you want to do workflow in ansible because it is pragmatic18:00
inc0because it is pragmatic18:00
sdakeif we ship 1.0.0 with a dep on ansible we are locked into ansible forever18:00
inc0but workflow in ansible != being locked to it18:00
sdake(if the workflow is designed in ansible)18:00
inc0no...18:00
*** zhubingbing_ has quit IRC18:00
kfox1111sdake: inc0 this is where we get into workflow=religious, and I wanted microservices so that people can pick workflow that they want.18:00
inc0I just create files18:00
kfox1111if someone wants ansible workflow, great. heat workflow, great. pure helm, great.18:01
inc0we can provide one18:01
inc0on our own18:01
sdakei agree we should provide one18:01
inc0as long as it's optional and pragmatic18:01
sdakeand i disagree it should be ansible :)18:01
kfox1111thats the issue. are we going to all be able to agree on just one?18:01
inc0kfox1111: I'd stick to one in repo18:02
inc0(just one)18:02
inc0supplying multiple will make it night impossible to work with18:02
kfox1111inc0: or 0.18:02
*** MasterOfBugs has joined #openstack-kolla18:02
sdakeour architecture spec indicates a helm + operators apprroach18:02
kfox1111inc0: yeah, but agreeing to one might be impossible.18:02
inc0no, we need quick start18:02
inc0kfox1111: we agreed on helm right?;)18:02
kfox1111thus requiring, a kolla like repo split.18:02
kfox1111kolla-kubernetes-helm, kolla-kubernetes-ansible, etc.18:03
inc0negative, requiring someone to take the job of translating ansible to helm18:03
sdakekfox1111 sounds good to me :)18:03
inc0then we drop ansible18:03
kfox1111and kolla-kubernetes is all the shared stuff.18:03
inc0it's as simple as that18:03
sdakeansible is fire and forget, hence it doesn't work well for managing openstack clouds18:03
inc0if we have full helm, I don't see value on keeping ansible18:04
sdakeit works well for deployment upgrade and other forms of management, I'd like to take that a step further18:04
inc0I'm working with ansible now because it takes 30min to create new role18:04
inc0and port it from kolla-ansible18:04
inc0and I'm lazy person18:04
inc0so I'm all for quick wins;)18:04
kfox1111ok, so the agreed thing to do for now, is18:04
*** eaguilar has joined #openstack-kolla18:05
kfox1111copy templates from kolla-ansible -> kolla-kubernetes to reduce deps, and allow us to set new, saner for kolla-kubernetes defaults.18:05
*** jtriley has joined #openstack-kolla18:05
kfox1111step 2, convert config to helm.18:05
kfox1111?18:05
inc0and simpler workflow to start with kolla-k8s18:05
inc0yes18:05
*** bjolo has quit IRC18:05
kfox1111what workflow?18:05
inc0only question is, who and when will work on step 218:05
inc0well, quickstart18:05
inc0git clone kolla-kubernetes, run this command, run that command, you have openstack18:06
rwellumworkflow = deployment or development inc0 ? (or both).18:06
inc0rwellum: both, development as testing our stuff18:07
kfox1111so, there's kinf of 4 workflow tools either being discussed seriously, or already exists.18:07
kfox11111. shell scripts18:07
kfox11112. helm+kubernetes-entrypoint18:07
kfox11113. kubernetes-operators18:07
kfox11114. ansible18:07
kfox11111 & 2 already exist.18:08
inc0yeah but shell is not great tool for the job18:08
kfox11113 is being prototyped.18:08
kfox1111do we really want 4?18:08
inc0and 2 is not complete - lacks config18:08
*** matrohon has quit IRC18:08
kfox1111I agree with shell not being good for end users.18:08
kfox1111that 2 or 3 is mroe for that.18:08
inc0so prototype - we'll see how it goes18:09
kfox1111well, we're talkign about dealing with config. does that need to influence everythign else though?18:09
inc0entrypoint + helm native configmaps is best case scenerio I see now18:09
kfox1111once we get helm packages for configmap, 2 & 3 can work natively with configmaps.18:09
sbezverk_kfox1111: why would we need 4??18:09
kfox1111sbezverk_: I'm arguing we dont.18:09
kfox1111sbezverk_: inc0 is talking about having it, at least termporarily?18:09
inc0ok, nobody wants that really18:09
sbezverk_kfox1111: +1 we do not18:09
inc0no18:09
inc0ansible roles end when configmaps are created imho18:10
kfox1111inc0: ah. ok.18:10
inc0and this is really really simple job to do for ansible18:10
kfox1111I'm ok with that then.18:10
*** jascott1_ has joined #openstack-kolla18:10
kfox1111well, let me ask something first...18:10
kfox1111ansible takes template, spits out etc/kolla/...18:10
kfox1111not sure we need etc/kolla/.... -> configmap to k8s18:11
kfox1111kubectl create configmap can do that.18:11
inc0well, it can, true18:11
kfox1111so we talking about tjust the first part, or all the parts of configmap jen?18:11
inc0but we need shell to iterate over the thing;)18:11
inc0so we need something anyway18:11
kfox1111yeah. but shell is trivial to understand in that.18:11
kfox1111its like a 3 line bash script.18:12
sdakekfox1111 for you shell is trivial for inc0 ansible is trivial18:12
inc0if that's true, why kollakube even exists?18:12
kfox1111loop over dirs, kubectl create configmap ..... ?18:12
openstackgerritMarcus Williams proposed openstack/kolla master: Add OpenDaylight Container to kolla  https://review.openstack.org/41636918:12
kfox1111inc0: because it did literally EVERYTHING before helm.18:12
sdakekfox1111 what i'm getting at is eveyrone has a different POV on triviality18:12
kfox1111sdake: I garantee the shell script will be at least 10 times smaller then the eqiv ansible.18:12
sbezverk_sdake: replacing shell by ansible just to act as bash has no value18:13
inc0kfox1111: alternatively instead of generating config files we can go straight to kube resources18:13
sdakesbezverk_ i didnt propose it no reason yell at me :)18:13
kfox1111inc0: +1 on that. as thats most of the work in getting things converted to helm.18:13
sbezverk_sdake: appologies, it was directed to inc018:13
kfox1111though a lot of work.18:13
inc0yeah18:14
inc0so let's not dig too deep into philosophical space of how bad is ansible vs shell18:14
inc0and just write damn thing;)18:14
sbezverk_inc0: we have been doing it almost  8 month, hav enot you noticed ;)18:15
inc0zuul is dead, long live the zuul :(18:15
*** sayantani01 has quit IRC18:15
jascott1_there is only zuul....18:15
*** pbandark has quit IRC18:16
*** adrian_otto has quit IRC18:16
kfox1111inc0: yeah, but still, the forloop in the gate, can be replaced with a forloop with kubectl create configmap with much less effort then adding ansible into the mix there.18:16
*** sayantani01 has joined #openstack-kolla18:16
*** adrian_otto has joined #openstack-kolla18:16
sbezverk_kfox1111: if we can have config files in chart folder, then we can do it using helm charts18:17
sbezverk_then we have consistent helm implementation for all aspects18:17
sdakeone thing we want to avoid is modifying charts to change configuration18:18
sdakecharts for config specifically are fine to modify18:18
sdakecharts for the implementaiton not ok18:18
kfox1111sbezverk_: +1 for helm charts eventaully. just saying, if the goal is to get rid of kollakube, we can do that by just switching configmap uploading to kubectl creat econfigmap trivially,18:19
sbezverk_sdake: I get this point, then we can push for a feature in helm --with-files18:19
sdakeconfig and implementation should be separate18:19
*** athomas has quit IRC18:19
kfox1111as thats all what kollakube does in that case.18:19
jascott1_sbezverk_ did someone say new helm feature ? :)18:19
sdakejascott1_ go forth and conquer :)18:19
sbezverk_jascott1_: yep18:20
jascott1_its been conquering me lately but I am whooping it as we speak18:20
kfox1111jascott1_: yeah. really looking forward to the parent chart feature. should really help. :)18:20
kfox1111hoping they merge it soon. :)18:20
sbezverk_jascott1_: currently chart rendering happens in tiller which has no visibility to clients file system, hence cannot include files located not in the chart's folder18:20
openstackgerritMarcus Williams proposed openstack/kolla master: Add OpenDaylight Container to kolla  https://review.openstack.org/41636918:21
jascott1_kfox1111 i had to fix the parent wins thing (which i did) but should have it all ready today.. release will be April fools day so not long to wait :)18:22
sbezverk_jascott1_: there was a proposal sometime ago to add --with-files key to attach other files18:22
jascott1_sbezverk_ i will hunt it down and check it18:22
sbezverk_jascott1_: I think having this feature could simplify a lot config/configmap generation thing18:23
sbezverk_jascott1_: cool, appreciate it18:23
openstackgerritKevin Fox proposed openstack/kolla-kubernetes master: Simple configmap to raw kubectl  https://review.openstack.org/44926218:25
jascott1_sbezverk_ this one? https://github.com/kubernetes/helm/issues/175418:25
kfox1111jascott1_: awesome. :)18:25
kfox1111inc0: I think that review ---^ will do most of the remaining work of getting rid of kollakube.18:26
*** sayantani01 has quit IRC18:26
sbezverk_jascott1_: yep18:26
openstackgerritMarcus Williams proposed openstack/kolla master: Add OpenDaylight Container to kolla  https://review.openstack.org/41636918:26
kfox1111no ansible needed for it.18:26
jascott1_cool, bookmarked18:26
*** sayantani01 has joined #openstack-kolla18:26
*** jmccarthy has joined #openstack-kolla18:28
*** yangyapeng has joined #openstack-kolla18:29
*** yangyapeng has quit IRC18:34
*** jmccarthy has quit IRC18:34
*** manheim has joined #openstack-kolla18:36
*** manheim has quit IRC18:36
*** lucasxu has quit IRC18:38
kfox1111is zuul broked?18:40
inc0zuul is dead18:40
inc0long live the zuul18:40
*** yangyapeng has joined #openstack-kolla18:42
*** jascott1_ has quit IRC18:42
sean-k-mooneyquick dumb question. is there an apache contiener? or does kolla install apache on the host18:42
kfox1111apache is in the containers that use it.18:43
*** unicell has joined #openstack-kolla18:43
sean-k-mooneykfox1111: so horizon and keysone have seperate apache instances?18:44
*** jtriley has quit IRC18:45
kfox1111sean-k-mooney: yes.18:45
kfox1111greatly simplifies the config, and allows you to restart one without effeting the other.18:46
*** yangyapeng has quit IRC18:47
sean-k-mooneyone of my deployments was scanned by our intenal vulnerbility system and has the sweet32 vlunerblity in the ssl implnetaion of the service running on port 44318:48
sean-k-mooneyim trying to figure out where i need to fix it18:48
*** jtriley has joined #openstack-kolla18:48
kfox1111not sure how its done in kolla-ansible, but I'm guessing its ssl terminated at an haproxy?18:50
openstackgerritMarcus Williams proposed openstack/kolla master: Add OpenDaylight Container to kolla  https://review.openstack.org/41636918:51
sean-k-mooneythis is a sandbox of our ci so i dont know if they installed haproxy but ill check both18:52
sean-k-mooneyhum no ha proxy and for some reason they have also instlled apach2 on the host. ssl is disabled for kolla on the sandbox it seams18:55
*** egonzalez has joined #openstack-kolla18:55
mnaserkolla-core: im going over common centos and oraclelinux failure and they seem to be related to a bug in the old docker releases18:56
inc0how old mnaser ?18:56
mnaseri proposed this a while back and it passed everything... could we get some traction? https://review.openstack.org/#/c/446158/18:56
mnaserhttps://github.com/docker/docker/issues/2833618:57
mnaserhttp://logs.openstack.org/71/446771/7/check/gate-kolla-dsvm-deploy-oraclelinux-source-centos-7-nv/3ddff5e/logs/kolla/nova/nova-scheduler.txt.gz#_2017-03-23_16_16_36_55618:57
mnaserhttp://logs.openstack.org/71/446771/7/check/gate-kolla-dsvm-deploy-oraclelinux-source-centos-7-nv/3ddff5e/logs/docker.txt.gz#_Mar_23_16_16_3618:57
mnaserlook at the time corrolation there18:57
inc0mnaser: replied to change18:58
mnaserinc0 good call, also fyi - http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22has%20become%20an%20orphan%2C%20killing%20it%5C%2218:59
mnaserso a lot of containers get killed in kolla jobs bc of this18:59
*** shashank_t_ has quit IRC18:59
mnaserinc0 actually shouldnt that be done in kolla-ansible re: your comment19:00
inc0mnaser: why?19:01
mnaserbecause no roles in the kolla repo?19:01
inc0ahh19:01
inc0right...19:01
mnaserill do that regardless tho19:01
mnaseras another review19:01
inc0yeah, I'm so newton19:01
mnaser:P19:01
inc0right, ofc you're right19:01
mnasersame19:01
mnaseri was with you too19:01
mnaserbut i was like "wait a second why cant i find this... search isnt returning anything.."19:02
openstackgerritMohammed Naser proposed openstack/kolla-ansible master: Update Docker to 1.13.1  https://review.openstack.org/44928519:04
mnaserbtw, im working off of this to try and identify more gate failures, just search this in logstash:19:05
mnaserbuild_status:"FAILURE" AND build_name:gate-kolla-dsvm* AND message:"Job complete"19:05
mnaserif anyone finds that useful19:05
egonzalezmnaser the new docker version is not having issues with libvirt-selinux?19:05
mnaseregonzalez i think as part of that review, i ran setenforce 0 in the gate19:06
mnaserthe justification was 1) kolla doesnt support selinux in the first place - docs say so19:06
mnaser2) that only means it worked "by luck" before19:06
mnaserthere is a comment there about it19:07
openstackgerritKevin Fox proposed openstack/kolla-kubernetes master: WIP: Fernet Token Support  https://review.openstack.org/44627419:08
*** deadnull has quit IRC19:09
sdakekfox1111 sweet i can abandon my review with the kollakube res fix solution19:09
*** shashank_t_ has joined #openstack-kolla19:09
kfox1111sdake: which review?19:10
sdakekfox1111 i did look at that review - and it seems wrong in some way19:10
egonzalezmnaser added a comment in kolla-ansible PS19:10
kfox1111sdake: which review?19:10
sdakekfox1111 https://review.openstack.org/#/c/449262/119:10
sean-k-mooneyinc0: did you ever get a chance to follow up on the osic apt repo issue?19:11
kfox1111sdake: wrong how so?19:11
sean-k-mooneyinc0: im currently patch my teams systems and just taught of it since im runing apt like 50 times in the last hour19:11
sdakethere is still kollakube still used ?19:11
kfox1111sdake: like I keep telling inc0, kollakube still does more then genconfig->configmap stuff.19:12
kfox1111not all things have been helmized yet.19:12
sdakekfox1111 i see19:12
kfox1111thats why getting completely out of kollakube is nontrivial.19:12
egonzalezinc0 mnaser not sure if pin point to docker-engine version, maybe is better remove the pin point?19:13
kfox1111some ceph stuff needs converting. some configmap stuff, keepliave and openvswitch-set-ip, and at least a secret I think.19:13
sdakekfox1111 the thing that triggered all this tlak of ansible is this review: https://review.openstack.org/#/c/439740/19:13
*** mgoddard has quit IRC19:13
mnaseregonzalez cool thanks19:13
sdakekfox1111 kollakube is broken today19:13
mnaserim not sure, i mean.. i think its better to pin things just in case docker breaks thingss19:14
mnaserso our gate can remain relatively healthy19:14
kfox1111sdake: how does it not work today? its owrking in the gate.19:14
mnaserbut i intend to poke this up to the 17.03 release soon if things are good19:14
sean-k-mooneymnaser: docker has broken us in the past so pinning is good provided we also test with new versions and update promptly when required19:15
*** krtaylor has joined #openstack-kolla19:15
kfox1111sdake: and the ps you put up breaks everything. so how does it work?19:15
sdakekfox1111 it breaks the gate, however works when run locally19:15
kfox1111I think its to try and elminnate the need for a symlink, but it doesn't seem to work?19:15
mnasersean-k-mooney i agree with that.  we can slowly but surely poke it up and try to keep up to date19:15
sdakekfox1111 if you run kollakube withoutsymlinks in place, it fials19:15
sdakeread the patch you will underwtaand why19:15
kfox1111sdake: gate's shouldn't be different then local. so why does it break?19:16
sdakei dont use symlinks locally19:16
sdakei pip install19:16
kfox1111sdake: I did. but its not clear why it breaks things to me.19:16
kfox1111do we not pip install in the gate?19:16
kfox1111I thought we did.19:16
openstackgerritMohammed Naser proposed openstack/kolla-ansible master: Update Docker to 1.13.1  https://review.openstack.org/44928519:17
sdakepip install -e19:17
sdakei pip install19:17
sdakepip install -e only works with symlinks19:17
mnaseregonzalez ty for that catch ^19:17
sdakepip install works in any condition19:17
sdakealthough kollakube is busted19:17
mnaseri think eventually the setup code in kolla/ should just rely on kolla-ansible for the deploy jobs19:18
mnaserbut thats for another patch19:18
kfox1111hmm....19:18
kfox1111so, we pip install kolla-kubernetes, but not kolla-ansible in the gate it looks like.19:18
*** shardy is now known as shardy_afk19:18
kfox1111sdake: so, in your ps, if you remove the added symlink, and add a pip install in here: https://github.com/openstack/kolla-kubernetes/blob/master/tools/setup_gate_common.sh#L8019:18
kfox1111I think it would fix the gate and fix your concern?19:18
egonzalezmnaser yep, makes sense, but is not as easy because deploy gates in kolla also build the images before deploy19:19
sdakepossible19:20
*** shashank_t_ has quit IRC19:20
sdakekfox1111 may require more gate work19:20
kfox1111sdake: possibly.19:20
kfox1111but we can't commit somethign that breaks the gate.19:20
kfox1111so either we recommend people just add the symlink,19:20
kfox1111or we fix the gate.19:20
sdakei agree19:20
*** shashank_t_ has joined #openstack-kolla19:20
sdakei'm not asking for it to be committed with a red gate19:21
sdakei'm pointing out what triggered the ansible discussion19:21
kfox1111k. yeah. I'm just saying what I think might fix the review.19:21
sdakeyup - atm working on https://review.openstack.org/#/c/447356/ - as i committed to have this ready to go by friday19:23
*** shashank_t_ has quit IRC19:23
kfox1111ah. ok. cool.19:23
*** adrian_otto has quit IRC19:24
sbezverk_sdake: the recheck you pushed for my ps came back all green..19:24
sbezverk_sdake: please take a look at ps when you have a chance19:25
sdakegerrit pokey atm19:26
sdakecan't do anytthing with it really19:26
*** adrian_otto has joined #openstack-kolla19:26
*** vhosakot has joined #openstack-kolla19:27
*** matrohon has joined #openstack-kolla19:32
sbezverk_sdake: ok did not know that..19:34
*** adrian_otto has quit IRC19:34
kfox1111gerrits getting worse. :/19:41
*** Margin has joined #openstack-kolla19:41
*** Margin has joined #openstack-kolla19:42
Marginhello19:43
*** g3ek has quit IRC19:43
*** shashank_t_ has joined #openstack-kolla19:51
*** g3ek has joined #openstack-kolla19:52
*** vhosakot has quit IRC19:54
*** vhosakot_ has joined #openstack-kolla19:55
*** rabel_b1 has quit IRC20:04
*** eaguilar has quit IRC20:06
*** mliima has quit IRC20:10
*** manheim has joined #openstack-kolla20:10
inc0can I get reviews plz? https://review.openstack.org/#/c/448258/20:11
openstackgerritKevin Fox proposed openstack/kolla-kubernetes master: Simple configmap to raw kubectl  https://review.openstack.org/44926220:12
kfox1111inc0: so, you want to merge it in seperate chunks?20:13
inc0yes, please20:13
inc0one role at the time20:13
inc0lol it took 3hrs for gates to pass20:13
kfox1111how long do you think it will take to remove the need for genconfig entirely?20:15
inc0how many patchsets? # of services20:16
inc07 or 8?20:16
inc0and then last patch to remove genconfig from gates and docs20:17
kfox1111no, like, wall clock?20:17
inc0took me 30min to get ks up20:17
inc0so if I'm alone, entirely possible to do it next week fully20:17
inc0with help, less20:17
kfox1111cause while its enabled in the gate, we have a mismatch between what the gate is testing, and what usuers will probably do.20:17
kfox1111I guess if its just for a week, then its not that risky, and gets ut there quicker?20:19
inc0quicker than what?20:19
kfox1111quicker then waiting until all ps's are ready and merging all at once?20:20
inc0please let's nt do that20:20
kfox1111It feels a little dirty having the gate half and half for how it gen's config.20:20
kfox1111though we did the same with helm. but that was really messy. :/20:20
inc0map-reduce;)20:21
inc0that allows others to work with me20:21
kfox1111I'm on the fence. but just slightly on the side of lets just get it done quickly and get it overwith.20:21
inc0quickest way - one role at time20:21
kfox1111+2'ed.20:21
inc0because again, allows cooperation of multiple people20:21
inc0thank you20:21
*** egonzalez has quit IRC20:21
*** satyar has quit IRC20:21
kfox1111well, with the current aproach, it will cause merge conflicts20:22
inc0keystone is almost ready too, zuul is finishing last gate20:22
inc0all other green20:22
kfox1111cause they will race on adding rm -rf /etc/kolla/xxxx's.20:22
kfox1111until they are all done with genconfig.20:22
inc0it's easy rebase tho20:22
inc0if needed at all20:22
kfox1111yeah.20:22
kfox1111not a major problem.20:23
kfox1111more of a problem if it wasn't like 8 things.20:23
*** sayantani01 has quit IRC20:24
kfox1111inc0: https://review.openstack.org/#/c/441550/20:24
kfox1111inc0: ironic totally working, and tested in gate. sbezverk did a huge amount of work there. great stuff. :)20:24
sdakeyup pretty impressive sbezverk_ ^^20:25
inc0120 patchsets :O20:27
inc0that might be record20:27
sdakeironic is really difficult to implement20:28
inc0I know20:28
sdakesbezverk_ should get a gold medal in the triathalon for getting that working and better yet getting it gating20:28
inc0kfox1111: https://review.openstack.org/#/c/449254/20:29
kfox1111gota head out for a little while.20:29
inc0keystone genconfig20:29
inc0doing nova now20:29
inc0sbezverk_: mind taking a look?20:29
inc0https://review.openstack.org/#/c/448258/20:29
kfox1111inc0: don't have time to look yet,20:29
kfox1111but just cursory,20:29
inc0np20:29
kfox1111we don't need the fernet/ssh stuff.20:30
inc0kk20:30
kfox1111doing it a much different way.20:30
inc0I removed20:30
kfox1111k8s native.20:30
inc0fernet stuff20:30
kfox1111ok. gotta go.20:30
inc0have fun20:30
rwellumsdake: sorry the free time I thought I'd have at 4 EST never happened - stuck doing something else.20:32
sdakerwellum roger - understood20:32
sdakerwellum perhaps tomorrow20:32
rwellumyes20:32
*** sayantani01 has joined #openstack-kolla20:36
sbezverk_inc0: done, good start !!20:42
inc0https://review.openstack.org/#/c/449254/ <- this is ks20:43
inc0green at first try;)20:43
*** jrobinson has joined #openstack-kolla20:44
sbezverk_inc0: I think kfox1111 doing fernet separately. Maybe we need to remove it from this ps?20:45
inc0sbezverk_: yeah, mind adding a comment?20:46
sbezverk_inc0: sure thing20:46
krtaylorinc0, sdake - (different topic) at altanta, I agreed to go look for help with the ansible extensible strategy plugin, of course, I then got pulled into 10 other things20:46
krtaylorinc0, sdake - but I did talk to our openstack-ansible guru20:47
inc0and?:)20:47
*** rwallner has quit IRC20:47
krtaylorinc0, sdake - he said "After looking at this a bit…it doesn’t seem like Kolla *needs* to put this change directly into Ansible.  Ansible is pluggable and Kolla could simply configure Ansible to use this strategy plugin versus putting it up in Ansible proper."20:48
inc0btw krtaylor https://github.com/inc0/ansible/commit/beba49876974d05c38d98ec2ec855bf59eb480e2 <- this is my prototype20:48
inc0krtaylor: yes and no20:48
inc0yes, we can put it there20:48
inc0but this file has to be GPLv3 licensed20:48
inc0and if part of project is GPLv3 licensed...you know the rest20:49
krtaylorah, I knew there was a catch20:49
inc0it's legal not techincal20:49
krtaylorI couldn't remember why20:49
openstackgerritMerged openstack/kolla-kubernetes master: WIP Move config generation to kolla-k8s  https://review.openstack.org/44825820:49
inc0now I see that commit msg wasn't ideal20:50
inc0well...what's done is doen20:50
krtaylorwell, the bad news is I couldn't find an ansible person to drive it20:50
krtaylorsorry it took so long to follow up20:50
inc0this might be documentation problem and create external project, licensed by gplv320:50
inc0that will have literally one file20:51
krtaylorheh, brute force approach, was there any objection to including it in ansible proper?20:52
inc0nothing much, but also not much was discussed/looked at20:52
inc0it's really fully optional and adds value20:52
inc0all is needed is some love and care20:53
*** manheim has quit IRC20:53
krtaylorI wish I had the bandwidth to take it on right now, but also not sure about if I can with the GPL20:54
kfox1111inc0: the plugin can be gplv3 compatable, and only gets covered in process.20:54
kfox1111and I believe the apache lisense is very explicitly v3 compatible.20:54
kfox1111so, should be fine?20:54
krtaylormingling code would make it viral I believe (not a legal expert)20:55
inc0well...ML didnt give me clear answer20:55
kfox1111viral in process.20:55
kfox1111gpl's a distribution license.20:55
inc0I'm positive that this file has to be gpl20:55
kfox1111I'm pretty sure it doesn't.20:56
inc0not sure about other files in same repo20:56
inc0it does, it imports gpl code20:56
inc0from very internals of ansible20:56
inc0that's the problme20:56
sdakekrtaylor thanks for getting back to me20:57
krtaylorI wish I had better news20:57
kfox1111stil wouldn't be dirivitive until it is in process.20:57
inc0kfox1111: I don't know and I'm not comfortable to navigate this field without guidance:/20:57
inc0too much gray and too big of a risk20:57
kfox1111inc0: fair enough.20:57
kfox1111I'm not a layer. but I have read through a lot of open source licenses. in detail.20:58
*** adrian_otto has joined #openstack-kolla20:58
inc0kfox1111: with a bit of pressure we can make you a layer20:59
kfox1111-120:59
kfox1111:)20:59
kfox1111I can parse machine code really well. I don't want to parse human code for a living.20:59
kfox1111I only do becausse I need to do so I can write machine code. :)20:59
inc0that's exactly my thing20:59
krtayloris there openstack legal that could assist?21:00
kfox1111yeah, shoudl be able to drop legal mailing list an email.21:00
inc0I think I did21:01
*** jascott1_ has joined #openstack-kolla21:01
kfox1111sbezverk_: don't think a recheck will clar that.21:01
kfox1111I think we need to -workflow + workflow.21:01
kfox1111cause it was in the merge check, not the normal check.21:02
sbezverk_kfox1111: what then :( ??21:07
sbezverk_I see21:07
jascott1_kfox1111 hit a little snag, how important is it to have the ability to merge values from two different modules into one parent key?21:08
jascott1_modules=charts21:08
*** krtaylor has quit IRC21:08
*** lamt has quit IRC21:10
*** lamt has joined #openstack-kolla21:12
jemcevoyI am having an issue with a new install at a customer site... When I run "pip install kolla" I get version 4.0.0 and there is no globals.yml file in the /usr/share/kolla/etc_examples/kolla ???  What am I do incorectly?21:12
jascott1_hi jemcevoy! might try checkout from git?21:15
*** kolla-slack1 has joined #openstack-kolla21:18
*** jascott1_ has quit IRC21:20
*** jascott1_ has joined #openstack-kolla21:20
*** kolla-slack1 has quit IRC21:21
*** jtriley has quit IRC21:22
*** kolla-slack has joined #openstack-kolla21:22
*** kolla-slack has quit IRC21:22
*** kolla-slack1 has joined #openstack-kolla21:23
*** kolla-slack has joined #openstack-kolla21:24
sdakejemcevoy the config files are in kolla-ansible deliverable21:25
*** jascott1_ has quit IRC21:25
*** jascott1_ has joined #openstack-kolla21:25
*** kolla-slack has quit IRC21:26
*** kolla-slack has joined #openstack-kolla21:26
*** eaguilar has joined #openstack-kolla21:26
*** matrohon has quit IRC21:27
sdakeslackbot do something21:28
*** lamt has quit IRC21:28
sdakeslackbot do something21:29
kolla-slack<sdake> sweet21:29
*** jascott1_ has quit IRC21:30
*** jascott1_ has joined #openstack-kolla21:32
openstackgerritMichal Jastrzebski (inc0) proposed openstack/kolla-kubernetes master: Move keystone configuration to kolla-k8s  https://review.openstack.org/44925421:33
*** pbandark has joined #openstack-kolla21:34
*** jascott1_ has quit IRC21:35
jemcevoysdake:  What is the pip command to get kolla-ansible deliverable ( I am not a pythonista )21:35
inc0jemcevoy: pip install kolla-ansible :)21:36
jemcevoyDo I need both kolla and kolla-ansible packages?21:37
inc0not really21:37
sdakekolla is for buildling21:37
sdakekolla-ansible is for deploying21:37
sdakeify ou wantot build your own or custom images you need kolla package21:37
sdakeif your happyto pull them form dockerhub that works too21:37
*** lucasxu has joined #openstack-kolla21:40
jemcevoyI fill local repository with the binary downloads.  But I think it build the docker images to fill the repo right?21:40
*** jascott1_ has joined #openstack-kolla21:42
jemcevoysdake:  This customer has one of those man-in-the-middle http/https proxies so the internet is kinda slow.  So I will need both to build the local repo for the cluster install I assume.21:43
inc0jemcevoy: well, not on same node tho;)21:44
inc0like, no need to be on same node21:44
kolla-slack<jascott1> that koala is naked!21:45
kolla-slack<jascott1> neeked!21:45
sdakeindeed jascott121:45
kolla-slack<jascott1> wow the integration seems to work21:45
*** adrian_otto has quit IRC21:47
openstackgerritMichal Jastrzebski (inc0) proposed openstack/kolla-kubernetes master: Move keystone configuration to kolla-k8s  https://review.openstack.org/44925421:47
jemcevoyinc0:  Not following... I have a new 12 node cluster and one ops server  on a older reclaimed system where I plan to run the kolla-ansible from to do the Newton install.  This ops server will also have cobbler to kickstart the new cluster...21:48
*** yangyapeng has joined #openstack-kolla21:48
jemcevoydo I install both kolla and kolla-ansible on the ops machine21:48
inc0what I'm saying, there is no need for them to be on same machine21:48
inc0it's entirely up to you21:48
inc0kolla will provide private registry of images, kolla-ansible will know how to deploy from it21:49
jemcevoyI installed both with pip... Guess I am good to go then I see that the globals.yml is there... I had to have them generate a "real" x509 cert for my repository since the self signed one did not work in their "secure" environment...21:51
*** eaguilar has quit IRC21:52
*** jascott1_ has quit IRC21:52
*** yangyapeng has quit IRC21:52
*** jascott1_ has joined #openstack-kolla21:53
*** eaguilar has joined #openstack-kolla21:56
*** jascott1_ has quit IRC21:57
kolla-slack<sdake> @jascott1 ya its pretty cool - surprised it was so easy to setup21:58
*** shashank_t_ has quit IRC22:00
*** jascott1_ has joined #openstack-kolla22:06
openstackgerritMerged openstack/kolla-kubernetes master: Virtualbmc and deployment of baremetal VM using ironic  https://review.openstack.org/44155022:06
dmsimardinc0, sdake: you nice people see anything out of the ordinary for RDO repos recently ? yum errors etc ?22:18
sdakedmsimard gate seems a little flakey but not certain22:19
fooliounokfox1111: you there?22:19
sdakedmsimard our gate was broken by other things22:19
sdakedmsimard not sure if it rdo was a problem or not22:19
kolla-slack<inc0> Oh great now I'll get ping on phone from irc22:19
kolla-slack<inc0> Just what I dreamt of22:19
sdakeinc0 use the mute feature22:19
sdake"mute the channel" in slack22:20
kolla-slack<inc0> I know :)22:20
kolla-slack<inc0> No that's actually good22:20
fooliounosdake: I see that you are setting the kolla_controller and kolla-compute to the same node. Will it work if I apply them to two separate nodes. Has it been tested :)22:20
sdakemany irc clients have various noises they can emit if you set it up22:20
sdakefooliouno tested no idea22:20
sdakefooliouno with ceph I believe so22:21
sdakefooliouno but not certain22:21
sdakefooliouno those docs you are referencing have not been tested however22:21
sdakefooliouno so YMMV22:21
sdakefooliouno please leave comments in them22:21
sdakefooliouno so they can be improved22:21
fooliounosdake: I will add comments as I run through them. If I use "host" configuration, then I cannot separate the two functions?22:22
*** kolla-slack1 has quit IRC22:23
sdakefooliouno ya its not ideal22:24
fooliounook .. thanks22:24
*** kolla-slack has quit IRC22:24
*** kolla-slack has joined #openstack-kolla22:25
*** mbruzek has quit IRC22:29
*** fooliouno has quit IRC22:31
*** eaguilar has quit IRC22:32
*** dave-mcc_ has joined #openstack-kolla22:32
*** dave-mccowan has quit IRC22:36
*** dave-mccowan has joined #openstack-kolla22:38
*** lucasxu has quit IRC22:39
openstackgerritMarcus Williams proposed openstack/kolla master: Add OpenDaylight Container to kolla  https://review.openstack.org/41636922:40
*** dave-mcc_ has quit IRC22:40
*** manheim has joined #openstack-kolla22:43
*** unicell1 has joined #openstack-kolla22:46
*** unicell has quit IRC22:46
*** vhosakot_ has quit IRC22:50
*** tonanhngo has quit IRC22:51
sdakev1k0d3n re your personal attack in the loci review, i have a right to review anything in the community in any way as I see fit.  My commentary was valid and is likely still valid.  Please don't attack other people personally - that doesn't fly in OpenStack.  TIA22:54
openstackgerritMarcus Williams proposed openstack/kolla master: Add OpenDaylight Container to kolla  https://review.openstack.org/41636922:57
*** tonanhngo has joined #openstack-kolla22:58
*** srwilkers has quit IRC22:58
*** g3ek has quit IRC23:00
*** tonanhngo has quit IRC23:02
*** Pavo has joined #openstack-kolla23:08
*** g3ek has joined #openstack-kolla23:09
*** Pavo has quit IRC23:11
v1k0d3nsdake please stop. nobody was attacking anyone personally at all.23:12
*** Margin has quit IRC23:15
*** fooliouno has joined #openstack-kolla23:15
*** Margin has joined #openstack-kolla23:16
kolla-slack<vhosakot> nice… hellor kolla friends in IRC :slightly_smiling_face:23:17
kolla-slack<vhosakot> I’m sending this from k8s slack :slightly_smiling_face:23:17
sdakevhosakot we see that ;)23:18
kolla-slack<vhosakot> awesome! :slightly_smiling_face:23:18
*** manheim has quit IRC23:18
spsuryamorning all23:19
*** MasterOfBugs has quit IRC23:19
sdakesup spsurya23:19
*** Margin has quit IRC23:20
*** pbandark has quit IRC23:31
*** adrian_otto has joined #openstack-kolla23:42
jemcevoyinc0:  I have a new issue.   kolla-build is failing on the customer site because docker build does not have the http/https man-in-the-middle proxy configured inside the centos7  containers so the rpm --import https://<blah> fails because it needs access to the ca.crt file for the proxy.   Any ideas on how to get around this?23:45
*** goldyfruit has quit IRC23:45
*** egonzalez has joined #openstack-kolla23:48
openstackgerritMohammed Naser proposed openstack/kolla-ansible master: Update Docker to 1.13.1  https://review.openstack.org/44928523:52

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!