15:59:12 <inc0> #startmeeting kolla
15:59:13 <openstack> Meeting started Wed Oct 25 15:59:12 2017 UTC and is due to finish in 60 minutes.  The chair is inc0. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:59:14 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
15:59:16 <openstack> The meeting name has been set to 'kolla'
15:59:35 <rwellum> o/
15:59:35 <inc0> #topic w00t
15:59:36 <inc0> o/
15:59:37 <duonghq> o/ evening guys
15:59:40 <gema> o/
15:59:45 <spsurya__> w00t o/
15:59:51 <inc0> duonghq: these topics in agenda - that's you?
15:59:55 <egonzalez> w00t/
16:01:02 <duonghq> inc0, yup
16:01:29 <duonghq> I forgot put my name on it
16:01:30 <duonghq> sorry
16:01:38 <inc0> no worries
16:02:40 <coolsvap> o/ w00t
16:03:12 <jascott1> woot?
16:03:17 <inc0> woot.
16:03:28 <inc0> #topic Announcements
16:04:05 <chason> woot.
16:04:08 <inc0> we have gates now! but we're still working on scenarios
16:04:20 <sbezverk> o/
16:05:09 <inc0> I encourage everyone to experiment with zuul v3 - now you can add gate in same patch
16:05:24 <britthouser> 0/
16:05:46 <inc0> #topic New policy yaml file
16:06:08 <inc0> forgot about community announcements, anyone had any?
16:06:17 <inc0> (sorry :()
16:06:48 <chason> inc0 Is there any sample commit about experiment with zuul v3 we can follow?
16:06:48 <inc0> guess not, duonghq go ahead
16:07:10 <duonghq> thank inc0
16:07:17 <rwellum> chason: +1
16:07:34 <inc0> #link https://review.openstack.org/#/c/513855/ https://review.openstack.org/#/c/512779/ https://review.openstack.org/#/c/513971/
16:08:40 <duonghq> Due to policy.json is migrating to policy.yaml (along with progress to move policy and doc into code), my colleague implemented the deployment of policy.yaml for aodh (for example)
16:08:45 <duonghq> #link https://review.openstack.org/#/c/514565/
16:09:08 <duonghq> We need review on this before implement for all other projects
16:09:47 <inc0> can't we just copy approach we have in keystone?
16:10:13 <chason> inc0 thanks! :)
16:10:47 <duonghq> inc0,  sorry I don't get your idea
16:11:08 <inc0> well, I don't know this issue too well, but I'll review it
16:11:19 <inc0> and get back to you
16:11:47 <duonghq> inc0,  the issue is described more detail in the bp
16:11:52 <duonghq> #link https://blueprints.launchpad.net/kolla-ansible/+spec/support-custom-policy-yaml
16:13:03 <inc0> duonghq, anyone, anything else in this topic?
16:13:17 <duonghq> inc0,  no, that's all for this topic
16:13:22 <inc0> ok
16:13:37 <inc0> #topic Keystone upgrade mechanism
16:13:39 <inc0> shoot
16:13:48 <duonghq> ya, me again,
16:14:00 <duonghq> I conducted some test on this
16:14:12 <duonghq> first: in my test, the upgrade is done well,
16:14:20 <duonghq> that is the good news
16:14:23 <duonghq> the bad news is
16:15:07 <duonghq> ah, sorry, I should put the link
16:15:10 <duonghq> #link https://review.openstack.org/#/c/425446/
16:15:28 <vhosakot> o/ (sorry I'm late)
16:15:40 <duonghq> okay, the bad news is if we upgrade whole system, the downtime taken by mariadb upgrade is not so small
16:15:57 <rwellum> o/ vhosakot
16:15:58 <duonghq> and it'll affect every service (which uses the mariadb)
16:16:11 <vhosakot> o/ rwellum :)
16:16:20 <inc0> mariadb rolling upgrade is not working? that's what you're saying?
16:16:32 <duonghq> if we upgrade only keystone, it take roughly 11sec downtime (in my testbed)
16:16:40 <egonzalez> With current mariadb Mariadb shouldn't have downtime
16:16:46 <duonghq> inc0, no, it's my observation
16:17:03 <inc0> yeah, currently mariadb doesn't seem to be addected
16:17:10 <inc0> affected*
16:17:18 <inc0> mariadb locks db during upgrade?
16:17:42 <duonghq> hmm, when mariadb upgrade task takes  place, keystone log show that it cannot connect to the db
16:18:55 <duonghq> I upgrade from P to master
16:19:09 <inc0> hmm, we need to test that
16:19:53 <duonghq> a little information about my testbed: 2VM with 4GB RAM each
16:20:06 <duonghq> VMs run on KVM
16:20:06 <inc0> ah, that might be it
16:20:18 <inc0> I barely got working openstack on 4gb ram
16:20:20 <duonghq> I cannot conduct 3 nodes test
16:20:34 <inc0> 8gb is minium I'd suggest on control node
16:21:45 <duonghq> ah, interesting, I'll try to increase the memory to 6GB RAM/VM, I'm not sure if my PC happy with 8GB RAM/VM
16:22:02 <duonghq> thask
16:22:04 <duonghq> thanks
16:22:26 <inc0> yeah when I deployed on 4gig it wasn't even working properly
16:23:01 <duonghq> got it
16:23:23 <inc0> but let's test it and confirm it
16:23:31 <inc0> anything else on this topic?
16:23:34 <duonghq> no, thanks
16:23:47 <inc0> #topic open discussion
16:24:02 <inc0> anyone? rwellum wanna share your libvirt issue?
16:24:22 <rwellum> Yes. It's broken in Pike. :(
16:24:44 <inc0> my theory is that it'll be broken in ocata too if you'd use tarballs ocata
16:24:46 <gema> We have started testing kolla-k8s on x86-64 to establish a baseline. I
16:24:49 <gema> asked kevinz to do this using the patch we are reviewing
16:24:51 <gema> (https://review.openstack.org/#/c/473588/). He is following the
16:24:54 <gema> (https://docs.openstack.org/kolla-kubernetes/latest/deployment-guide.html).
16:24:57 <gema> deployment guide
16:24:58 <gema> )
16:25:00 <gema> (sorry
16:25:04 <egonzalez> Is broken in 5.0.0 fixed in 5.0.1 no released yet
16:25:14 <inc0> egonzalez: broken how?
16:25:23 <rwellum> Yes inc0 have a couple of avenues to go.
16:25:35 <egonzalez> Broken in current tag, but was fixed and backported
16:25:49 <inc0> ok, but what was issue? user in qemu.conf?
16:25:49 <egonzalez> The centos 7.4 released changed libvirt version
16:26:19 <rwellum> egonzalez: I think you're talking about: https://review.openstack.org/#/c/513546/1
16:27:01 <rwellum> FYI - I patched that and it did not help
16:27:10 <inc0> yeah I don't think it's this issue
16:27:20 <inc0> error rwellum is seeing is about CA certificates
16:27:40 <egonzalez> Yeah but was merged in other change https://review.openstack.org/#/c/506529/8/ansible/roles/nova/templates/qemu.conf.j2
16:27:52 <inc0> btw what's your docker version rwellum ?
16:27:57 <rwellum> 1.12
16:28:03 <inc0> :S
16:28:33 <inc0> ok, we'll debug more
16:28:38 <rwellum> 1.12.6 actually
16:28:42 <inc0> rwellum: mind sharing whole error msg?
16:29:17 <rwellum> When I remove the tcp_listen code I get a different error though - not sure which is relevant
16:29:31 <inc0> well might be
16:29:57 <inc0> anyway, let's move to regular kolla channel and contirnue debugging
16:30:09 <gema> I got an update, and a question
16:30:12 <gema> before we go
16:30:15 <inc0> shoot
16:30:26 <gema> as I stupidly pasted before
16:30:32 <inc0> (I was writing question if anyone else have anything to say;))
16:30:42 <gema> kevinz has started ramping up and reviewing the patch you told us last week
16:30:52 <gema> So far we have found an issue in the documentation when running (kollakube res create configmap mariadb), do we raise this as a bug or shall we put it in the comments of the patch?
16:31:12 <gema> essentially, what do we do with the feedback how do we help fixing things?
16:31:18 <inc0> gema: so we're in process of revisiting whole guide - over weekends;)
16:31:22 <gema> ok
16:31:28 <gema> we can contribute to that then
16:31:36 <inc0> kevinz is in Asia right?
16:31:45 <gema> yep, shanghai
16:32:01 <inc0> ok, I'll jump in in his timezone and talk to him
16:32:11 <gema> ok
16:32:17 <gema> the other thing he has observed
16:32:19 <inc0> about plans we have and what we want to achieve
16:32:28 <gema> bugs: (mariadb) https://bugs.launchpad.net/kolla-kubernetes/+bug/1708610 and (rabbitmq) https://bugs.launchpad.net/kolla-kubernetes/+bug/1723590
16:32:29 <openstack> Launchpad bug 1708610 in kolla-kubernetes "mariadb pods failed to start" [Undecided,New]
16:32:30 <openstack> Launchpad bug 1723590 in kolla-kubernetes "RabbitMQ Pod fails with CrashLoopBackOff" [Undecided,New]
16:32:37 <gema> he hits those two
16:32:49 <gema> again, want the feedback on the patch or want us to help implement fixes or..
16:33:08 <gema> he can workaround the issues by following instructions on the bugs
16:33:15 <inc0> sure, thanks
16:33:28 <EmilienM> hey
16:33:44 <EmilienM> have you folks moved to native zuul v3 jobs? or still use legacy ones?
16:34:04 <inc0> EmilienM: still in process of moving fully, but yeah
16:34:08 <EmilienM> ok
16:34:09 <inc0> we don't use legacy any more
16:34:12 <EmilienM> second question
16:34:16 <EmilienM> who is in Sydney
16:34:32 <EmilienM> I sent an email about discussing the LTS + stable policy tag for operators
16:34:40 <EmilienM> it would be cool to get someone from Kolla as well.
16:34:53 <inc0> Jeffrey4l would be best person
16:34:55 <rwellum> EmilienM: I am
16:35:02 <EmilienM> ok, please join then
16:35:15 <EmilienM> I'm done :P
16:35:31 * EmilienM disappears
16:35:35 <rwellum> gema: I worked with kevinz last night (for me) - I think his mariadb issues was user error
16:35:38 <inc0> hehe;) thanks EmilienM for this cameo;) if you need any help with zuulv3, I'd be happy to
16:35:54 <gema> rwellum: ok, I am glad he is syncing with you
16:35:58 <EmilienM> inc0: I will
16:36:07 <gema> rwellum: so you think it is not really a bug?
16:36:19 <gema> (not one that needs to be fixed in kolla-k8s anyway)
16:36:39 <rwellum> gema: well, he'd started the mariadb container, before openvswitch - in the guide you start the other way.
16:36:52 <rwellum> Maybe relevant - not sure - but asked him to try anyway
16:37:06 <rwellum> But if he's using the playbook - maybe it is an issue
16:37:10 <gema> yeah, he got further I think and then he encountered bug 1708610
16:37:11 <openstack> bug 1708610 in kolla-kubernetes "mariadb pods failed to start" [Undecided,New] https://launchpad.net/bugs/1708610
16:37:50 <rwellum> Ah ok
16:38:16 <gema> I am just trying to get my head around whether we should be putting this feedback on the patch we are testing and tie it all up, or whether we should be working on the different bugs we are finding along the way
16:38:20 <gema> what is the preferred option
16:38:38 <inc0> gema: so right now just force your way through process
16:38:45 <gema> inc0: which is?
16:38:47 <inc0> next step is to automate it in ansible
16:39:39 <inc0> yeah, that's a good question;) just deploy one service after another until all is deployed
16:39:50 <gema> inc0: if you want me to choose I will make kevin update the patch he is testing and then try to fix the bugs he is encountering
16:40:00 <gema> with different patches
16:40:12 <gema> because they seem to be outstanding issues
16:40:21 <rwellum> gema: +1 then document it all - we can review
16:40:33 <rwellum> I'm not seeing those two issues though
16:41:14 <gema> rwellum: that's interesting, I will ask him to sync with you interms of comparing environments
16:41:56 <rwellum> yup
16:42:04 <gema> inc0, rwellum:that's all from me, happy to continue as agreed
16:42:13 <inc0> thank you
16:42:22 <inc0> anyone else? we have 15min
16:42:35 <chason> Me! I'd like to talk about the docs work.
16:42:36 <rwellum> thanks - glad that kevinz is onboard  - you promised someone and delivered. :)
16:42:48 <inc0> chason: shoot
16:42:51 <chason> I have started the bp https://blueprints.launchpad.net/kolla/+spec/optimize-the-documentation-format
16:42:54 <gema> wait for the patches rwellum  ;)
16:43:11 <chason> And have proposed two commits(need review). If we look them good, I will start the same work of kolla-ansible. :D
16:43:34 <inc0> cool, thanks, I'll take a look after meeting
16:43:34 <rwellum> gema: :)
16:43:52 <inc0> (sorry my reviews are slacking, zuulv3 took most of my time over last 2 weeks)
16:44:02 <rwellum> When does RST drop?
16:46:01 <inc0> rwellum: I don't understand question
16:46:35 <chason> There is much content exists in kolla-ansible need to be revised. So anyone who wants to join this work is welcomed. :P
16:47:08 <rwellum> Looking at the bp - RST becomes the only acceptable format soon - was wondering when that is.
16:47:27 <rwellum> "After we upgrading the sphinx ( 1.6.2) and openstackdocstheme (1.17.0), the current writing styles are no longer applicable. We need to update our docs to standard RST conventions."
16:48:03 <chason> #link https://etherpad.openstack.org/p/kolla-queens-doc-restructure
16:49:08 <rwellum> chason: missing kolla-kubernetes ?
16:50:21 <chason> Oh, we missed it!
16:50:23 <chason> Sorry about that.
16:50:44 <chason> I will add kolla-k8s to our list. :D
16:50:48 <chason> Thanks
16:51:01 <rwellum> ty
16:51:09 <inc0> thanks chason rwellum
16:51:11 <rwellum> Less documentation so won't be as bad
16:51:23 <inc0> anyone else wants to discuss something?
16:52:00 <inc0> guess not:)
16:52:03 <inc0> thank you all!
16:52:09 <inc0> #endmeeting kolla