15:01:52 <mgoddard> #startmeeting kolla
15:01:53 <openstack> Meeting started Wed Aug  5 15:01:52 2020 UTC and is due to finish in 60 minutes.  The chair is mgoddard. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:01:54 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
15:01:56 <openstack> The meeting name has been set to 'kolla'
15:01:58 <mgoddard> #topic rollcall
15:02:01 <mgoddard> \o
15:02:42 <priteau> o/
15:02:47 <yoctozepto> o/
15:04:11 <mnasiadka> o/
15:04:12 <mgoddard> #topic agenda
15:04:20 <mgoddard> * Roll-call
15:04:22 <mgoddard> * Announcements
15:04:24 <mgoddard> ** Kolla kall tomorrow
15:04:26 <mgoddard> * Review action items from last meeting
15:04:28 <mgoddard> * CI status
15:04:30 <mgoddard> * Victoria release planning (kayobe)
15:04:32 <mgoddard> * Victoria release planning (kolla & kolla ansible)
15:04:34 <mgoddard> * Kolla klub and kall
15:04:36 <mgoddard> #topic announcements
15:04:46 <mgoddard> #info Kolla kall tomorrow
15:06:09 <mgoddard> #info mgoddard talked about Kayobe & Kolla: https://youtu.be/0liqSO0SZ60?t=4842
15:06:25 <mgoddard> virtual open infra meetup london/manchester
15:06:32 <mgoddard> Any others?
15:07:16 <mgoddard> #topic Review action items from last meeting
15:07:22 <mgoddard> mgoddard to message openstack-discuss about focal & victoria upgrade
15:07:24 <mgoddard> I did not
15:07:39 <mgoddard> I was going to earlier, but thought I'd wait until after we discuss it today
15:07:46 <mgoddard> #topic CI status
15:07:55 <bbezak> o/
15:07:58 <mgoddard> we have a few issue
15:08:06 <mgoddard> Kolla
15:08:08 <mgoddard> Train: gitdb issue https://review.opendev.org/#/c/744501
15:08:09 <patchbot> patch 744501 - kolla (stable/train) - requirements.txt: pin gitdb(2) versions - 2 patch sets
15:08:14 <mgoddard> Stein: ironic-conductor broken: https://27734da6eed83e608576-333f8b549856196606ae2476154ba69d.ssl.cf1.rackcdn.com/743923/1/gate/kolla-build-centos-source/10d084d/kolla/build/000_FAILED_ironic-conductor.log
15:08:28 <mgoddard> hrw: around?
15:10:16 <mgoddard> I guess not
15:11:12 <mgoddard> mnasiadka: did you look into that ironic issue any further?
15:11:40 <mnasiadka> mgoddard: nope, can do tomorrow
15:11:58 <mgoddard> ok
15:12:16 <mgoddard> #action mnasiadka to look into kolla stein ironic-conductor failure
15:12:22 <mgoddard> thanks
15:12:36 <mgoddard> otherwise I think we are mostly ok
15:13:28 <mgoddard> #topic Victoria release planning (kayobe)
15:14:04 <mgoddard> Looks like it's time to give mnasiadka access to kayobe +2
15:14:17 <yoctozepto> congrats mnasiadka
15:14:22 <mnasiadka> Yay
15:14:29 <yoctozepto> :D
15:15:38 <mgoddard> added
15:16:12 <priteau> Just in time for his holidays :)
15:16:17 <mgoddard> priteau, mnasiadka, dougsz: anything to say about kayobe this week?
15:16:49 <dougsz> Nothing from me this week
15:17:05 <priteau> Not really. I am slowly emerging from work on another project, hopefully more time for upstream in August
15:17:15 <mgoddard> Release still too far away to be busy :)
15:17:21 <mgoddard> (on kayobe)
15:17:42 <mgoddard> I have some performance improvements in the works
15:17:58 <mgoddard> first some easy wins
15:18:05 <mgoddard> then some less easy wins
15:18:13 <mgoddard> but hopefully all wins
15:18:38 <mgoddard> One thing we do need to do this cycle is the configuration walkthrough in docs
15:19:00 <mgoddard> is it something we should collaborate on, or just have one person do it?
15:20:33 <mgoddard> tough crowd
15:20:36 <mgoddard> Let's move on
15:20:46 <mgoddard> #topic Victoria release planning (kolla & kolla ansible)
15:21:17 <priteau> Sorry, I was in another conversation
15:21:37 <mgoddard> priteau: no problem
15:21:44 <priteau> About previous topic, I guess if one person could do a draft to start with
15:22:00 <priteau> Then we can review and improve collaboratively
15:22:12 <mgoddard> makes sense
15:22:24 <yoctozepto> priteau: am I seeing a volunteer? :D
15:23:19 <priteau> If I have the time
15:24:07 <mgoddard> ok, back to kolla
15:24:23 <mgoddard> was hoping hrw would be here to discuss focal
15:24:36 <yoctozepto> perf improvements going nicely thanks to mgoddard (and me a bit)
15:24:37 <mgoddard> mnasiadka: I think you touched it recently. What's the status?
15:25:00 <yoctozepto> https://review.opendev.org/#/q/project:openstack/kolla-ansible+branch:master+topic:bp/performance-improvements
15:25:05 <dcapone2004> is it meeting time?  I lost power and just got it back, so I missed any replies to my questions from yesterday and any activity for that matter...
15:25:17 <yoctozepto> dcapone2004: meeting running
15:25:57 <yoctozepto> also a poc fix for the nasty restart-too-much bug: https://review.opendev.org/744207
15:25:57 <patchbot> patch 744207 - kolla-ansible - [WIP] Fix looped notifies and optimize further - 4 patch sets
15:26:02 <mgoddard> dcapone2004: http://eavesdrop.openstack.org/irclogs/%23openstack-kolla/
15:26:23 <yoctozepto> it has some impact on perf, makes some things better, some possibly worse
15:26:41 <mnasiadka> mgoddard: k-a change looks green
15:26:42 <priteau> dcapone2004: http://eavesdrop.openstack.org/irclogs/%23openstack-kolla/%23openstack-kolla.2020-08-04.log.html#t2020-08-04T22:58:18
15:26:58 <mnasiadka> mgoddard: I need to analyse services logs and other stuff, but should be at least semi-okay ;)
15:30:46 <mgoddard> mnasiadka: ok, guess we can push ahead with the kolla-ansible part?
15:31:41 <mnasiadka> mgoddard: yeah, seems like it
15:32:32 <mnasiadka> mgoddard: well, actually it has depends-on to kolla change
15:33:05 <mgoddard> mnasiadka: I think hrw wanted to reverse that
15:33:26 <mgoddard> although it wouldn't work due to distro_python_version
15:33:49 <mgoddard> I suppose we have a cyclic dependency
15:33:52 <mnasiadka> makes sense, that I just noticed that - needs some workaround - but I'll look into it
15:34:57 <mnasiadka> mgoddard: we might need to move to python3 instead of python{{ distro_python_version }} in wsgi's at least
15:35:37 <mgoddard> mnasiadka: I seem to remember the stuff in the wsgi config isn't actually used
15:35:43 <mgoddard> so we might be able to remove it
15:36:26 <mnasiadka> mgoddard: well, keystone fernet also fails on python{{ distro_python_version }}
15:36:43 <mnasiadka> This one is also easy to remove - either way I'll follow up
15:36:49 <mgoddard> #action mgoddard to message openstack-discuss about focal & victoria upgrade
15:36:55 <mgoddard> still need to do that I guess
15:37:00 <mgoddard> ok
15:38:01 <mgoddard> yoctozepto: so this handlers change
15:38:15 <mgoddard> we have a few things to consider
15:38:23 <mgoddard> backporting
15:38:25 <mgoddard> performance
15:38:45 <mgoddard> having spent a lot of time improving performance recently I don't really want to go backwards
15:39:03 <yoctozepto> I am inclined to merge the two tasks into one
15:39:11 <yoctozepto> the filtered loops seems to be improving perf a lot
15:39:23 <yoctozepto> the only hit left would be the include_ indirection
15:39:30 <yoctozepto> but ansible kinda forces that on us
15:39:33 <mgoddard> it makes a small difference at scale
15:39:37 <yoctozepto> the alternative is actually less performant
15:39:41 <yoctozepto> due to no loop filtering
15:40:09 <mgoddard> tasks hurt performance far more than skips
15:40:14 <yoctozepto> so wdyt about merging tasks (to avoid skip and extra task) and pursuing this approach for all?
15:40:21 <mgoddard> and the include method adds multiple tasks
15:41:38 <mgoddard> plus the overhead of the include task
15:42:02 <mgoddard> it's probably more overhead than I have saved in total so far :)
15:42:54 <yoctozepto> mgoddard: for neutron the new approach is more performant at my scale
15:43:08 <mgoddard> what is your scale?
15:43:16 <yoctozepto> 20-30
15:43:24 <yoctozepto> this is because the filtering helped A LOT
15:43:34 <yoctozepto> as neutron has many undeployed services
15:43:43 <yoctozepto> so the total amount of tasks went down
15:43:46 <mgoddard> it will help with an include for sure
15:43:49 <yoctozepto> but it is a win for neutron
15:43:58 <yoctozepto> not every service out there
15:44:08 <yoctozepto> but most maybe hit less than you are afraid
15:44:13 <yoctozepto> I agree to merge the tasks
15:44:19 <yoctozepto> one is just perfect for that
15:44:23 <yoctozepto> also easier to coordinate
15:44:35 <yoctozepto> but the include_ indirection must stay imho
15:44:41 <yoctozepto> as I see no other clean-enough approach
15:44:53 <yoctozepto> I wish ansible just fixed that handlers logic lol
15:45:41 <yoctozepto> I mean notifies* logic
15:45:43 <yoctozepto> but whatever
15:46:02 <mgoddard> +1 for merging the tasks
15:46:15 <yoctozepto> mgoddard: could you review https://review.opendev.org/743961 as it's prereq for that fix
15:46:15 <patchbot> patch 743961 - kolla-ansible - Performance: optimize genconfig - 4 patch sets
15:46:27 <yoctozepto> we would backport it like the other perf impv
15:46:31 <mgoddard> include vs not needs further thought
15:46:56 <yoctozepto> true that, but I can't imagine a solution without getting ansible fixed
15:47:02 <yoctozepto> might as well employ charms :-)
15:47:47 <mgoddard> makes sense
15:47:53 <yoctozepto> :D
15:48:39 <mgoddard> the other solution is handlers that might do nothing. It's not clean, but if it's faster we should go with it
15:49:06 <mgoddard> I'll propose it
15:49:26 <mgoddard> #topic Kolla klub and kall
15:49:31 <mgoddard> We have a kall tomorrow
15:49:39 <yoctozepto> ok, but you are getting the penalty of new horde of skipped tasks instead of filtered includes :-)
15:49:45 <mgoddard> Continuing the docs discussion
15:49:51 <yoctozepto> this is why I'm not imagining other solutions
15:49:56 <yoctozepto> ok, kall
15:50:09 <yoctozepto> yes, we started exploring kayobe last time
15:50:20 <mgoddard> I missed that part
15:50:25 <yoctozepto> I would like someone to chair that
15:50:27 <mgoddard> will be there tomorrow
15:50:29 <yoctozepto> we wrote it down for you
15:50:30 <mgoddard> I can do it
15:50:32 <yoctozepto> or, well, I did
15:50:40 <mgoddard> ok
15:50:56 <yoctozepto> thanks, happy to have kayobe head to chair kayobe discussion :-)
15:51:07 <yoctozepto> we'll try meetpad tomorrow again
15:51:19 <mgoddard> re kolla klub, I was wondering if we should break for the summer?
15:51:23 <yoctozepto> but on problems switch to google meeting
15:51:47 <mgoddard> attendance has been lower recently, maybe we've reached peak klub and need a break
15:52:00 <yoctozepto> re kolla klub - agreed, the times are weird due to the pandemic but people want to go holidaying
15:52:36 <mgoddard> was also thinking we should try another time for people in APAC, if there is interest
15:52:57 <mgoddard> could potentially run the same onboarding & update sessions with a different group
15:54:40 <yoctozepto> ++
15:55:12 <mgoddard> #topic Open discussion
15:55:16 <mgoddard> anyone have anything?
15:55:46 <yoctozepto> not me
15:57:14 <headphoneJames> Does the Nova API wsgi txn need any additional changes?
15:57:47 <mgoddard> headphoneJames: good question. Haven't looked lately
15:57:49 <headphoneJames> https://review.opendev.org/724794
15:57:50 <patchbot> patch 724794 - kolla-ansible - Add support for encrypting Nova API - 29 patch sets
16:01:05 <mgoddard> will try to look this week.
16:01:07 <mgoddard> thanks all
16:01:09 <mgoddard> #endmeeting