15:59:12 #startmeeting kolla 15:59:13 Meeting started Wed Oct 25 15:59:12 2017 UTC and is due to finish in 60 minutes. The chair is inc0. Information about MeetBot at http://wiki.debian.org/MeetBot. 15:59:14 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 15:59:16 The meeting name has been set to 'kolla' 15:59:35 o/ 15:59:35 #topic w00t 15:59:36 o/ 15:59:37 o/ evening guys 15:59:40 o/ 15:59:45 w00t o/ 15:59:51 duonghq: these topics in agenda - that's you? 15:59:55 w00t/ 16:01:02 inc0, yup 16:01:29 I forgot put my name on it 16:01:30 sorry 16:01:38 no worries 16:02:40 o/ w00t 16:03:12 woot? 16:03:17 woot. 16:03:28 #topic Announcements 16:04:05 woot. 16:04:08 we have gates now! but we're still working on scenarios 16:04:20 o/ 16:05:09 I encourage everyone to experiment with zuul v3 - now you can add gate in same patch 16:05:24 0/ 16:05:46 #topic New policy yaml file 16:06:08 forgot about community announcements, anyone had any? 16:06:17 (sorry :() 16:06:48 inc0 Is there any sample commit about experiment with zuul v3 we can follow? 16:06:48 guess not, duonghq go ahead 16:07:10 thank inc0 16:07:17 chason: +1 16:07:34 #link https://review.openstack.org/#/c/513855/ https://review.openstack.org/#/c/512779/ https://review.openstack.org/#/c/513971/ 16:08:40 Due to policy.json is migrating to policy.yaml (along with progress to move policy and doc into code), my colleague implemented the deployment of policy.yaml for aodh (for example) 16:08:45 #link https://review.openstack.org/#/c/514565/ 16:09:08 We need review on this before implement for all other projects 16:09:47 can't we just copy approach we have in keystone? 16:10:13 inc0 thanks! :) 16:10:47 inc0, sorry I don't get your idea 16:11:08 well, I don't know this issue too well, but I'll review it 16:11:19 and get back to you 16:11:47 inc0, the issue is described more detail in the bp 16:11:52 #link https://blueprints.launchpad.net/kolla-ansible/+spec/support-custom-policy-yaml 16:13:03 duonghq, anyone, anything else in this topic? 16:13:17 inc0, no, that's all for this topic 16:13:22 ok 16:13:37 #topic Keystone upgrade mechanism 16:13:39 shoot 16:13:48 ya, me again, 16:14:00 I conducted some test on this 16:14:12 first: in my test, the upgrade is done well, 16:14:20 that is the good news 16:14:23 the bad news is 16:15:07 ah, sorry, I should put the link 16:15:10 #link https://review.openstack.org/#/c/425446/ 16:15:28 o/ (sorry I'm late) 16:15:40 okay, the bad news is if we upgrade whole system, the downtime taken by mariadb upgrade is not so small 16:15:57 o/ vhosakot 16:15:58 and it'll affect every service (which uses the mariadb) 16:16:11 o/ rwellum :) 16:16:20 mariadb rolling upgrade is not working? that's what you're saying? 16:16:32 if we upgrade only keystone, it take roughly 11sec downtime (in my testbed) 16:16:40 With current mariadb Mariadb shouldn't have downtime 16:16:46 inc0, no, it's my observation 16:17:03 yeah, currently mariadb doesn't seem to be addected 16:17:10 affected* 16:17:18 mariadb locks db during upgrade? 16:17:42 hmm, when mariadb upgrade task takes place, keystone log show that it cannot connect to the db 16:18:55 I upgrade from P to master 16:19:09 hmm, we need to test that 16:19:53 a little information about my testbed: 2VM with 4GB RAM each 16:20:06 VMs run on KVM 16:20:06 ah, that might be it 16:20:18 I barely got working openstack on 4gb ram 16:20:20 I cannot conduct 3 nodes test 16:20:34 8gb is minium I'd suggest on control node 16:21:45 ah, interesting, I'll try to increase the memory to 6GB RAM/VM, I'm not sure if my PC happy with 8GB RAM/VM 16:22:02 thask 16:22:04 thanks 16:22:26 yeah when I deployed on 4gig it wasn't even working properly 16:23:01 got it 16:23:23 but let's test it and confirm it 16:23:31 anything else on this topic? 16:23:34 no, thanks 16:23:47 #topic open discussion 16:24:02 anyone? rwellum wanna share your libvirt issue? 16:24:22 Yes. It's broken in Pike. :( 16:24:44 my theory is that it'll be broken in ocata too if you'd use tarballs ocata 16:24:46 We have started testing kolla-k8s on x86-64 to establish a baseline. I 16:24:49 asked kevinz to do this using the patch we are reviewing 16:24:51 (https://review.openstack.org/#/c/473588/). He is following the 16:24:54 (https://docs.openstack.org/kolla-kubernetes/latest/deployment-guide.html). 16:24:57 deployment guide 16:24:58 ) 16:25:00 (sorry 16:25:04 Is broken in 5.0.0 fixed in 5.0.1 no released yet 16:25:14 egonzalez: broken how? 16:25:23 Yes inc0 have a couple of avenues to go. 16:25:35 Broken in current tag, but was fixed and backported 16:25:49 ok, but what was issue? user in qemu.conf? 16:25:49 The centos 7.4 released changed libvirt version 16:26:19 egonzalez: I think you're talking about: https://review.openstack.org/#/c/513546/1 16:27:01 FYI - I patched that and it did not help 16:27:10 yeah I don't think it's this issue 16:27:20 error rwellum is seeing is about CA certificates 16:27:40 Yeah but was merged in other change https://review.openstack.org/#/c/506529/8/ansible/roles/nova/templates/qemu.conf.j2 16:27:52 btw what's your docker version rwellum ? 16:27:57 1.12 16:28:03 :S 16:28:33 ok, we'll debug more 16:28:38 1.12.6 actually 16:28:42 rwellum: mind sharing whole error msg? 16:29:17 When I remove the tcp_listen code I get a different error though - not sure which is relevant 16:29:31 well might be 16:29:57 anyway, let's move to regular kolla channel and contirnue debugging 16:30:09 I got an update, and a question 16:30:12 before we go 16:30:15 shoot 16:30:26 as I stupidly pasted before 16:30:32 (I was writing question if anyone else have anything to say;)) 16:30:42 kevinz has started ramping up and reviewing the patch you told us last week 16:30:52 So far we have found an issue in the documentation when running (kollakube res create configmap mariadb), do we raise this as a bug or shall we put it in the comments of the patch? 16:31:12 essentially, what do we do with the feedback how do we help fixing things? 16:31:18 gema: so we're in process of revisiting whole guide - over weekends;) 16:31:22 ok 16:31:28 we can contribute to that then 16:31:36 kevinz is in Asia right? 16:31:45 yep, shanghai 16:32:01 ok, I'll jump in in his timezone and talk to him 16:32:11 ok 16:32:17 the other thing he has observed 16:32:19 about plans we have and what we want to achieve 16:32:28 bugs: (mariadb) https://bugs.launchpad.net/kolla-kubernetes/+bug/1708610 and (rabbitmq) https://bugs.launchpad.net/kolla-kubernetes/+bug/1723590 16:32:29 Launchpad bug 1708610 in kolla-kubernetes "mariadb pods failed to start" [Undecided,New] 16:32:30 Launchpad bug 1723590 in kolla-kubernetes "RabbitMQ Pod fails with CrashLoopBackOff" [Undecided,New] 16:32:37 he hits those two 16:32:49 again, want the feedback on the patch or want us to help implement fixes or.. 16:33:08 he can workaround the issues by following instructions on the bugs 16:33:15 sure, thanks 16:33:28 hey 16:33:44 have you folks moved to native zuul v3 jobs? or still use legacy ones? 16:34:04 EmilienM: still in process of moving fully, but yeah 16:34:08 ok 16:34:09 we don't use legacy any more 16:34:12 second question 16:34:16 who is in Sydney 16:34:32 I sent an email about discussing the LTS + stable policy tag for operators 16:34:40 it would be cool to get someone from Kolla as well. 16:34:53 Jeffrey4l would be best person 16:34:55 EmilienM: I am 16:35:02 ok, please join then 16:35:15 I'm done :P 16:35:31 * EmilienM disappears 16:35:35 gema: I worked with kevinz last night (for me) - I think his mariadb issues was user error 16:35:38 hehe;) thanks EmilienM for this cameo;) if you need any help with zuulv3, I'd be happy to 16:35:54 rwellum: ok, I am glad he is syncing with you 16:35:58 inc0: I will 16:36:07 rwellum: so you think it is not really a bug? 16:36:19 (not one that needs to be fixed in kolla-k8s anyway) 16:36:39 gema: well, he'd started the mariadb container, before openvswitch - in the guide you start the other way. 16:36:52 Maybe relevant - not sure - but asked him to try anyway 16:37:06 But if he's using the playbook - maybe it is an issue 16:37:10 yeah, he got further I think and then he encountered bug 1708610 16:37:11 bug 1708610 in kolla-kubernetes "mariadb pods failed to start" [Undecided,New] https://launchpad.net/bugs/1708610 16:37:50 Ah ok 16:38:16 I am just trying to get my head around whether we should be putting this feedback on the patch we are testing and tie it all up, or whether we should be working on the different bugs we are finding along the way 16:38:20 what is the preferred option 16:38:38 gema: so right now just force your way through process 16:38:45 inc0: which is? 16:38:47 next step is to automate it in ansible 16:39:39 yeah, that's a good question;) just deploy one service after another until all is deployed 16:39:50 inc0: if you want me to choose I will make kevin update the patch he is testing and then try to fix the bugs he is encountering 16:40:00 with different patches 16:40:12 because they seem to be outstanding issues 16:40:21 gema: +1 then document it all - we can review 16:40:33 I'm not seeing those two issues though 16:41:14 rwellum: that's interesting, I will ask him to sync with you interms of comparing environments 16:41:56 yup 16:42:04 inc0, rwellum:that's all from me, happy to continue as agreed 16:42:13 thank you 16:42:22 anyone else? we have 15min 16:42:35 Me! I'd like to talk about the docs work. 16:42:36 thanks - glad that kevinz is onboard - you promised someone and delivered. :) 16:42:48 chason: shoot 16:42:51 I have started the bp https://blueprints.launchpad.net/kolla/+spec/optimize-the-documentation-format 16:42:54 wait for the patches rwellum ;) 16:43:11 And have proposed two commits(need review). If we look them good, I will start the same work of kolla-ansible. :D 16:43:34 cool, thanks, I'll take a look after meeting 16:43:34 gema: :) 16:43:52 (sorry my reviews are slacking, zuulv3 took most of my time over last 2 weeks) 16:44:02 When does RST drop? 16:46:01 rwellum: I don't understand question 16:46:35 There is much content exists in kolla-ansible need to be revised. So anyone who wants to join this work is welcomed. :P 16:47:08 Looking at the bp - RST becomes the only acceptable format soon - was wondering when that is. 16:47:27 "After we upgrading the sphinx ( 1.6.2) and openstackdocstheme (1.17.0), the current writing styles are no longer applicable. We need to update our docs to standard RST conventions." 16:48:03 #link https://etherpad.openstack.org/p/kolla-queens-doc-restructure 16:49:08 chason: missing kolla-kubernetes ? 16:50:21 Oh, we missed it! 16:50:23 Sorry about that. 16:50:44 I will add kolla-k8s to our list. :D 16:50:48 Thanks 16:51:01 ty 16:51:09 thanks chason rwellum 16:51:11 Less documentation so won't be as bad 16:51:23 anyone else wants to discuss something? 16:52:00 guess not:) 16:52:03 thank you all! 16:52:09 #endmeeting kolla