15:00:30 <mgoddard> #startmeeting kolla
15:00:31 <openstack> Meeting started Wed Jun 19 15:00:30 2019 UTC and is due to finish in 60 minutes.  The chair is mgoddard. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:00:32 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
15:00:34 <openstack> The meeting name has been set to 'kolla'
15:00:47 <mgoddard> #topic rollcall
15:00:52 <mgoddard> \o
15:00:59 <scottsol> o/
15:01:06 <JamesBen_> \o
15:01:12 <gardenshed> o/
15:01:39 <mloza> \o/
15:02:46 <mgoddard> mnasiadka: around?
15:03:13 <mnasiadka> mgoddard: sort of :)
15:03:18 <mnasiadka> o/
15:03:19 <mgoddard> :)
15:03:23 <stackedsax> I thought I was an hour late -- just starting?
15:03:32 <mgoddard> hi stackedsax yeah
15:03:36 <mgoddard> #topic agenda
15:03:47 <mgoddard> * Roll-call
15:03:49 <mgoddard> * Announcements
15:03:51 <mgoddard> ** Last call for priorities https://etherpad.openstack.org/p/kolla-train-priorities
15:03:53 <mgoddard> * Review action items from last meeting
15:03:55 <mgoddard> * Kolla whiteboard https://etherpad.openstack.org/p/KollaWhiteBoard
15:03:57 <mgoddard> * Stein release status
15:03:59 <mgoddard> * Kolla Ansible performance https://etherpad.openstack.org/p/kolla-ansible-speed-ups
15:04:01 <mgoddard> * (vishwanathj): Elasticsearch
15:04:03 <mgoddard> * Octavia usability improvements
15:04:05 <mgoddard> * Vote: Kayobe to become a kolla deliverable?
15:04:07 <mgoddard> #topic announcements
15:04:09 <mgoddard> #info Last call for kolla train priorities
15:04:14 <mgoddard> #info Last call for kolla train priorities
15:04:21 <mgoddard> #link https://etherpad.openstack.org/p/kolla-train-priorities
15:04:35 <mgoddard> please cast your votes - no need to be core
15:04:52 <mgoddard> Any other announcements?
15:05:33 <mgoddard> #topic Review action items from last meeting
15:05:43 <mgoddard> mgoddard to try again again to disable ocata publishing job
15:05:51 <mgoddard> I finally did it!
15:06:20 <mgoddard> Turns out we needed to remove the job definitions on the stable/ocata branch, rather than using a branch filter on master
15:06:24 <mgoddard> #link https://review.opendev.org/665742
15:06:27 <mgoddard> Needs reviews
15:06:41 <mgoddard> And depends on https://review.opendev.org/665935
15:06:53 <mgoddard> #topic Kolla whiteboard https://etherpad.openstack.org/p/KollaWhiteBoard
15:07:43 <mgoddard> Checking CI status
15:08:59 <mgoddard> Mostly looks happy
15:10:38 <mgoddard> I went through and backported all bug fixes in the last month yesterday
15:10:38 <mgoddard> lots to review
15:10:38 <mgoddard> Anyone have anything else from the whiteboard?
15:10:52 <mgoddard> Any other features we should be trackng on there?
15:11:41 <mgoddard> Added CentOS/RHEL8
15:12:03 <mgoddard> #topic Stein release status
15:12:40 <mgoddard> We switched to Ceph nautilus using upstream ceph packages, but this caused issues for tripleo so now we are switching back to CentOS ceph packages
15:12:54 <mgoddard> However this is blocked on an update to those packages
15:13:37 <mgoddard> This is the one to watch:
15:13:41 <mgoddard> #link https://review.opendev.org/#/c/666015/
15:14:05 <mgoddard> Any questions on Stein?
15:14:36 <mgoddard> #topic Kolla Ansible performance https://etherpad.openstack.org/p/kolla-ansible-speed-ups
15:14:55 <mgoddard> #link https://etherpad.openstack.org/p/kolla-ansible-speed-ups
15:15:20 <mgoddard> This was added by klindgren who has been experiencing performance issues at a scale of around 400 hypervisors
15:15:32 <mgoddard> Who is interested in kolla-ansible at large scale?
15:15:41 <mnasiadka> Me
15:16:04 <mnasiadka> Do we have any ARA recordings of the run with 400 hypervisors?
15:16:11 <gardenshed> me
15:17:22 <mgoddard> mnasiadka: I'm not aware of any. I don't know if they'd be happy to share but we could ask
15:17:39 <mgoddard> I'd like to see results from running with the profile_tasks callback plugin
15:18:37 <mgoddard> I see a bunch of people on the Etherpad
15:18:56 <mgoddard> Does anyone have experience with running kolla ansible at scale? Or even just ansible?
15:19:30 <mnasiadka> Ansible yes, but no crazy huge roles
15:20:10 <mgoddard> If you have some good tips or ideas, please add to the etherpad
15:20:43 <mnasiadka> First we need to understand what actions are problematic (templating?)
15:21:05 <mgoddard> Yes, I think templating is hitting us hard
15:21:35 <mgoddard> There are certain patterns that kolla-ansible uses, and we should try to profile those in isolation
15:22:05 <mgoddard> I'd like to try to get interested parties to join a call at some point to discuss this
15:22:14 <mgoddard> scottsol, stackedsax - interested?
15:22:21 <scottsol> yes I can join that
15:22:49 <stackedsax> I'd like to focus on the TLS stuff for now
15:22:55 <mgoddard> stackedsax: sure
15:23:02 <scottsol> I can look over what we have done already and update the etherpad
15:23:51 <scottsol> its not "huge" scale yet but we already started to see some bottlenecks even with 20-30 nodes
15:24:17 <scottsol> just some minor changes have already started to cut it down
15:26:01 <mgoddard> that would be helpful
15:26:24 <mgoddard> let's keep it to the etherpad for now
15:26:56 <mgoddard> any other thoughts?
15:28:04 <mgoddard> #topic (vishwanathj): Elasticsearch
15:28:07 * jroll rolls in late, sorry
15:28:15 <mgoddard> hi jroll
15:29:13 <mgoddard> vishwanathj who proposed this topic is not here
15:29:25 <mgoddard> #topic Octavia usability improvements
15:29:31 <mgoddard> rm_work: around?
15:30:23 <jroll> I won't promise he's not, but he usually isn't this time of day :)
15:30:37 <mgoddard> This was something that came up at the Denver summit kolla feedback session - Octavia is quite hard to deploy in kolla and requires some manual effort
15:30:39 <mgoddard> #link https://etherpad.openstack.org/p/DEN-train-kolla-feedback
15:31:40 <openstackgerrit> Doug Szumski proposed openstack/kolla-ansible master: Enable InfluxDB TSI by default  https://review.opendev.org/666318
15:31:55 <mgoddard> From that pad:
15:31:56 <mgoddard> generate certificates is really manual
15:31:56 <mgoddard> doesn't support seperate CAs for internal / external
15:31:56 <mgoddard> Sounds useful to have
15:31:56 <mgoddard> OSA has some interesting bits
15:31:57 <mgoddard> For context, rm_work is Octavia PTL and also has kolla experience, so could be in a good position to help improve this situation
15:32:12 <stackedsax> thanks for the context on where that came from.
15:32:15 <mgoddard> scottsol and co have deployed octavia, and have some notes they could share
15:32:15 <stackedsax> rm_dev and I chatted because you told us to, mgoddard
15:32:36 <mgoddard> :)
15:32:51 <stackedsax> but now that I know the context, we might chat again
15:33:06 <mgoddard> sorry, thought I'd explained but maybe not well enough
15:33:40 <mgoddard> so what's the best path forward here?
15:33:43 <stackedsax> no worries, we had a nice convo, anway.  Just, maybe, not so useful :D
15:34:06 <mgoddard> rm_work suggested he might not have lots of time immediately, but I'm sure would be able to provide guidance
15:34:20 <stackedsax> yeah, that's the sense I got, as well
15:34:35 <scottsol> we could start by listing what has to be done manually? maybe another etherpad...?
15:35:01 <stackedsax> certainly, from that pad it seems like it's something that should be addressed within the scope of the TLS everywhere conversation
15:35:11 <scottsol> we have some changes that we can push back to fix some bugs / add minor improvments
15:35:14 <stackedsax> scottsol: that would be good
15:35:27 <stackedsax> maybe just on the TLS everywhere pad
15:35:35 <stackedsax> https://etherpad.openstack.org/p/kolla-internal-tls
15:35:40 <mgoddard> how about a ML discussion on openstack-discuss? starting with where we're at now, and then looking at gaps
15:35:55 <scottsol> sure
15:36:32 <mgoddard> we could always link to a pad from there, but the ML might be a good way to strike up a cross-project discussion
15:37:37 <mgoddard> I can kick that off. Interested parties please look out for it and reply
15:37:53 <mgoddard> #action mgoddard to post to ML about octavia in kolla
15:38:27 <mgoddard> other thoughts on this?
15:39:10 <mgoddard> #topic Vote: Kayobe to become a kolla deliverable?
15:39:15 <mgoddard> #link http://lists.openstack.org/pipermail/openstack-discuss/2019-June/006901.html
15:39:43 <mgoddard> ^ That post started the discussion about Kayobe becoming an official project
15:39:50 <mgoddard> There are two options on the table
15:39:56 <mgoddard> 1. become a deliverable of the Kolla project
15:40:00 <mgoddard> 2. become an official top level OpenStack project
15:40:56 <mgoddard> the only feedback it received from the kolla team was from hrw who was happy with either approach
15:41:34 <mgoddard> I think the best path forward is to take a vote. I would not participate
15:41:42 <mgoddard> Does that sound reasonable?
15:41:52 <scottsol> yep :)
15:42:02 <chason> agree :)
15:42:12 <mgoddard> I think the electorate would be the kolla core team, but would certainly be interested to hear opinions from others in the community
15:42:24 <mgoddard> mnasiadka?
15:42:43 <yoctozepto> o/
15:42:52 <yoctozepto> did not know you have a meeting today
15:43:07 <yoctozepto> not core but opting for 1
15:43:32 <yoctozepto> to me it's an extra layer of glue (kolla)
15:44:21 <scottsol> I think they would fit well together, and it would make it easier for keeping them sync'd. If not they could always be playing cat and mouse to stay aligned.
15:44:29 <mnasiadka> mgoddard: agree to vote, don’t have an opinion on kayobe atm :)
15:44:54 <mgoddard> ok. I will send an email to openstack-discuss
15:45:27 <mgoddard> #action mgoddard to ask openstack-discuss to vote on making kayobe a kolla deliverable
15:45:47 <mgoddard> mnasiadka: now is the time to form one then :)
15:46:18 <mgoddard> Any other thoughts on this?
15:46:51 <mgoddard> #topic Open discussion
15:47:09 <mgoddard> Anyone have anything else to discuss today?
15:47:43 <yoctozepto> mgoddard: I have one thing - CI errors
15:47:58 <mgoddard> yoctozepto: go ahead
15:47:58 <yoctozepto> if everybody was kind enough to report them to launchpad :-)
15:48:11 <yoctozepto> some are random due to Zuul and infra
15:48:18 <yoctozepto> but some are random but real
15:48:29 <yoctozepto> so I think it's worth the effort
15:48:41 <mgoddard> yoctozepto: good point
15:48:52 <yoctozepto> vide nova
15:48:58 <mgoddard> we are probably a bit too 'recheck happy'
15:49:14 <yoctozepto> yeah, figured that out
15:49:27 <mgoddard> I have thought about this in the past
15:50:03 <mgoddard> what about an etherpad with a taxonomy of CI errors and a count of how many times each has been hit?
15:50:09 <yoctozepto> also in this regard - scope of kolla-ansible testing - it does not try to deploty everything and surely it should not
15:50:19 <openstackgerrit> Merged openstack/kolla-ansible stable/rocky: Fix issue finding custom, host specific plugins  https://review.opendev.org/666143
15:50:24 <yoctozepto> > <mgoddard> what about an etherpad with a taxonomy of CI errors and a count of how many times each has been hit?
15:50:28 <yoctozepto> good idea
15:50:42 <mgoddard> yoctozepto: want to create one?
15:50:49 <yoctozepto> though if it looks legit then straight to launchpad
15:50:52 <mgoddard> and link to it from the whiteboard?
15:50:59 <mgoddard> +1
15:51:02 <yoctozepto> mgoddard: sure, need to learn how to :D
15:51:17 * yoctozepto writes it down in todo
15:51:27 <yoctozepto> > <yoctozepto> also in this regard - scope of kolla-ansible testing - it does not try to deploty everything and surely it should not
15:51:27 <mgoddard> just make up a name and append it to https://etherpad.openstack.org/p/
15:51:47 <yoctozepto> mgoddard: ok, I'll prepare it
15:51:48 <mgoddard> #action yoctozepto to create an etherpad with a taxonomy of CI errors and a count of how many times each has been hit
15:52:21 <mgoddard> we have a CI status section in the whiteboard which would be a good place for this link
15:52:38 <yoctozepto> (for the third time) <yoctozepto> also in this regard - scope of kolla-ansible testing - it does not try to deploty everything and surely it should not
15:52:47 <yoctozepto> but maybe it's time to rethink what should be
15:52:57 <yoctozepto> maybe it has already been done :P
15:53:18 <mgoddard> we have ongoing work to improve CI coverage
15:53:56 <mgoddard> normally as additional jobs
15:54:00 <mgoddard> is that what you mean?
15:54:42 <yoctozepto> yup
15:54:50 <mgoddard> just added the WIP patches to the whiteboard (at the bottom)
15:55:51 <yoctozepto> ok, cool
15:56:18 <mgoddard> I'd like to see the ironic and mariadb patches get some more attention
15:56:34 <mgoddard> as they're ready to go
15:56:43 <mgoddard> #link https://review.opendev.org/#/c/568829/
15:56:53 <mgoddard> #link https://review.opendev.org/655663
15:57:27 <mgoddard> there is also more coverage we could get within our existing jobs, e.g. horizon as discussed earlier
15:57:45 <yoctozepto> yup
15:58:00 <yoctozepto> I also saw something ceph-related today/yesterday?
15:58:08 <yoctozepto> where I suggested glance
15:58:25 <yoctozepto> https://review.opendev.org/#/c/666068/
15:58:29 <yoctozepto> 'tis the change
15:58:38 <mgoddard> that's true. It was adding the ceph-upgrade job to kolla
15:58:57 <yoctozepto> you did -1 on your workflow
15:59:31 <mgoddard> yoctozepto: yeah, I had some dummy changes to trigger the jobs. Need to remove now I know they work
15:59:59 <mgoddard> glance is an interesting idea. We should probably add to all ceph jobs for both kolla and kolla-ansible
16:00:17 <mgoddard> We're at time.
16:00:24 <mgoddard> Thanks for attending, everyone
16:00:29 <mgoddard> #endmeeting