15:00:18 <mgoddard> #startmeeting kolla
15:00:18 <openstack> Meeting started Wed Mar  4 15:00:18 2020 UTC and is due to finish in 60 minutes.  The chair is mgoddard. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:00:19 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
15:00:21 <openstack> The meeting name has been set to 'kolla'
15:00:24 <mgoddard> #topic rollcall
15:00:26 <mgoddard> \o
15:00:28 <scottsol> o/
15:00:35 <yoctozepto> o/
15:00:40 <osmanlicilegi> o/
15:00:40 <hrw> /o
15:00:58 <priteau> o/
15:01:05 <hrw> crowdy
15:03:13 <mgoddard> #topic agenda
15:03:24 <mgoddard> * Roll-call
15:03:26 <mgoddard> * Announcements
15:03:28 <mgoddard> * Review action items from last meeting
15:03:30 <mgoddard> * CI status
15:03:32 <mgoddard> * OpenDev PTG in Vancouver - will anyone attend?
15:03:34 <mgoddard> * Kolla SIG (aka Kolla Klub?) https://etherpad.openstack.org/p/kolla-sig
15:03:36 <mgoddard> * Discuss: Ansible may run more handlers than originally planned https://bugs.launchpad.net/kolla-ansible/+bug/1863510
15:03:37 <openstack> Launchpad bug 1863510 in kolla-ansible "Ansible may run more handlers than originally planned" [Medium,Triaged]
15:03:38 <mgoddard> * Time to drop CentOS 7 jobs?
15:03:40 <mgoddard> * Ussuri release planning (kayobe)
15:03:42 <mgoddard> * Ussuri release planning (kolla & kolla ansible)
15:03:44 <mgoddard> #topic announcements
15:03:46 <mgoddard> #info CI broken in myriad ways
15:03:49 <mgoddard> We'll cover that next
15:03:59 <mgoddard> Any others?
15:04:25 <hrw> CirrOS 0.5.0 released
15:04:33 <hrw> patch for k-a in gerrit
15:04:53 <mgoddard> #info CirrOS 0.5.0 released
15:04:56 <mgoddard> nice work hrw
15:05:00 <mgoddard> any new features?
15:05:46 <yoctozepto> can neutron finally stick to using cirros for their advanced networking testing instead of relying on ubuntu 14.04? :D
15:05:48 <hrw> 5.3 kernel, graphical console working on aarch64, lot of virtio included (rng as well)
15:06:19 <hrw> yoctozepto: you know that cirros is simple image to test does your cloud works?
15:06:30 <hrw> yoctozepto: convince 14.04 -> 18.04 move?
15:06:45 <yoctozepto> hrw: I just asked
15:07:06 <yoctozepto> ubuntu is bloated
15:07:13 <yoctozepto> no idea why cirros was not sufficient
15:07:28 <mgoddard> #topic Review action items from last meeting
15:07:30 <yoctozepto> let's move on
15:07:39 <mgoddard> dougsz to write bug report about nova SSH nproc issue
15:07:44 <mgoddard> dougsz: did you get to it?
15:07:58 <mgoddard> I didn't see it
15:08:54 <mgoddard> #action dougsz to write bug report about nova SSH nproc issue
15:08:59 <mgoddard> #topic CI status
15:09:06 <mgoddard> ok now for some fun
15:09:15 <mgoddard> we have a few issues at the moment
15:09:25 <mgoddard> and to top it off some mirrors seem to be down
15:09:48 <yoctozepto> yeah, today is fun all around
15:10:12 <mgoddard> let's start with kolla
15:10:28 <mgoddard> kolla.common.utils.tempest: ERROR: Package 'patrole' requires a different Python: 2.7.5 not in '>=3.6' (C7@master/Ussuri)
15:11:15 <yoctozepto> we can drop this from kolla after oslo gets fixed
15:11:23 <yoctozepto> because we need k-a working
15:11:35 <mgoddard> example: https://e3487461d10f0d2cc5e1-c866b68a342a1b98ad74b0accf39cca4.ssl.cf1.rackcdn.com/706042/2/check/kolla-build-centos-source/ae638c7/kolla/build/000_FAILED_tempest.log
15:12:12 <yoctozepto> sanity check: any idea why it hit master and not train?
15:12:45 <yoctozepto> we theoretically build train in c7
15:12:53 <yoctozepto> even on master branch
15:12:56 <mgoddard> have we done it today?
15:13:17 <yoctozepto> nope, not touched yet
15:13:25 <yoctozepto> in any dimension :-)
15:15:04 <mgoddard> do we understand the underlying issue here?
15:15:43 <mgoddard> tempest dropped py2 support?
15:16:51 <mgoddard> seems like it happened a while ago though
15:17:14 <yoctozepto> mhm, patrole dropped it
15:17:23 <yoctozepto> and we just take that branch
15:17:29 <yoctozepto> it's not from upper-constraints
15:17:37 <yoctozepto> hence it killed master c7
15:17:47 <mgoddard> ok
15:17:54 <mgoddard> it's centos7, we don't care too much
15:18:04 <mgoddard> but will it break stable branches?
15:18:11 <mgoddard> or will UC save us there?
15:18:34 <mgoddard> let's see tomorrows periodic
15:18:53 <mgoddard> I guess we either drop c7 or drop c7 tempest
15:19:21 <yoctozepto> drop c7 altogether
15:19:25 <hrw> drop c7
15:19:26 <yoctozepto> it's a mix-and-match
15:19:37 <mgoddard> it's coming up next
15:19:42 <mgoddard> ok, onto k-a
15:19:47 <yoctozepto> stables not affected because they pin to released version
15:19:50 <hrw> if someone want c7 then train supports it
15:20:13 <mgoddard> (2020-03-04) Ussuri 100% fail at keystone
15:20:20 <mgoddard> this is due to oslo.cache 2.1.0
15:20:32 <mgoddard> hberaud proposed a patch, we're testing it now
15:20:52 <mgoddard> https://review.opendev.org/#/c/711227/
15:20:52 <patchbot> patch 711227 - kolla - DNM: Test oslo.cache fix - 3 patch sets
15:21:09 <mgoddard> affects py3 jobs on master
15:21:16 <mgoddard> finally (I hope)
15:21:26 <mgoddard> ARA 0.x broken on py2
15:21:35 <mgoddard> I raised https://bugs.launchpad.net/kolla-ansible/+bug/1866079
15:21:37 <openstack> Launchpad bug 1866079 in kolla-ansible train "Stable branch CI fails when gathering facts - invalid syntax (fake_filesystem.py, line 1195)" [Critical,In progress] - Assigned to Mark Goddard (mgoddard)
15:21:47 <openstackgerrit> Dincer Celik proposed openstack/kolla-ansible stable/stein: Fixes gnocchi-api script name for Ubuntu/Debian  https://review.opendev.org/711249
15:21:58 <mgoddard> proposed a workaround for us https://review.opendev.org/711239
15:21:58 <patchbot> patch 711239 - kolla-ansible (stable/train) - CI: Pin pyfakefs to <4 for Python 2 - 1 patch set
15:22:08 <mgoddard> and a fix for ARA: https://review.opendev.org/711240
15:22:08 <patchbot> patch 711240 - recordsansible/ara (stable/0.x) - Fix ansible execution on Python 2 - 1 patch set
15:22:46 <mgoddard> oh, we missed one
15:23:15 <mgoddard> we're deploying master containers on triain
15:23:17 <mgoddard> https://launchpad.net/bugs/1866054
15:23:18 <openstack> Launchpad bug 1866054 in kolla-ansible train "Kolla Ansible train deploys master containers by default" [High,In progress] - Assigned to Mark Goddard (mgoddard)
15:23:55 <mgoddard> this actually means we're subject to the oslo.cache issue on train too (which is how yoctozepto caught it)
15:24:13 <yoctozepto> indeed
15:24:28 <yoctozepto> I say we merge your ara and fix to release
15:24:35 <yoctozepto> and approve
15:24:37 <yoctozepto> as one
15:24:47 <yoctozepto> it's not to backport
15:24:52 <yoctozepto> and reno is quite clear
15:25:07 <mgoddard> will need to backport ara
15:25:10 <mgoddard> can do that first
15:25:33 <yoctozepto> well, patches are never removed
15:25:41 <yoctozepto> so does not matter too much for order
15:25:56 <mgoddard> ok
15:26:04 <mgoddard> I think we are done with CI
15:26:07 <mgoddard> kayobe green \o.
15:26:21 <mgoddard> kolla cli red
15:26:25 <mgoddard> functional down due to ceph gone (yoctozepto)
15:26:43 <mgoddard> I saw you were pushing kolla-cli patches yoctozepto, was it for that?
15:27:04 <yoctozepto> mgoddard: yeah, sorry, forgot to remove it
15:27:18 <mgoddard> nice and green
15:27:20 <mgoddard> ok, onwards
15:27:30 <yoctozepto> mgoddard: gmann proposed py2.7 drop in k-c
15:27:37 <yoctozepto> mgoddard: and it got stuck
15:27:52 <mgoddard> hmm
15:28:00 <yoctozepto> mgoddard: hence needed unblock
15:28:11 <yoctozepto> mgoddard: k-c sees no continued work
15:28:17 <mgoddard> indeed
15:28:41 <mgoddard> feel free to ping interested parties rather than fix yourself
15:28:56 <mgoddard> #topic OpenDev PTG in Vancouver - will anyone attend?
15:29:07 <hrw> nope
15:29:17 <yoctozepto> mgoddard: I have no idea who is interested in here :D
15:29:23 <yoctozepto> not going myself
15:29:31 <hrw> #1 budget, #2 coronavirus corpo rules
15:29:32 <yoctozepto> too expensive
15:29:34 <mgoddard> Once again I was omitted from the mailing list asking which projects will attend Vancouver. Thankfully priteau noticed
15:29:50 <yoctozepto> hmph, no love for kolla :/
15:30:00 <hrw> Vancouver is nice city
15:30:07 <mgoddard> to be fair, kolla has shown no love for ptg recently :)
15:30:45 <mgoddard> we haven't really had productive development discussions in person for $years
15:30:49 <hrw> was kolla at any PTG after Dublin?
15:30:56 <mgoddard> it's good for ops feedback etc
15:31:08 <mgoddard> but that's summit not PTG
15:31:19 <hrw> mgoddard: we need european ptg to make it cheaper
15:31:35 <mgoddard> maybe, but that just makes it expensive for everyone else :)
15:31:37 <yoctozepto> next is Berlin I think?
15:31:41 <yoctozepto> well
15:31:56 <osmanlicilegi> I may attend Vancouver (love my company)
15:32:30 <mgoddard> osmanlicilegi: great. Just be prepared that there will probably be no official kolla discussions
15:32:40 <mgoddard> I'm sure it will still be interesting and useful though
15:32:52 <mgoddard> I think a virtual PTG works well for us
15:32:57 <mgoddard> and works well for our planet too
15:33:02 <osmanlicilegi> +1 for vptg
15:33:03 <yoctozepto> +1
15:33:28 <hrw> +1
15:33:42 <hrw> and we can couch as much as we want ;D
15:34:36 <mgoddard> seems we have a rough consensus - no official presence from kolla, but there may be some koalas present for informal discussions
15:34:45 <mgoddard> I'll pass it on
15:35:13 <mgoddard> Gonna mix up the agenda order a little, keep you on your toes
15:35:15 <mgoddard> #topic Time to drop CentOS 7 jobs?
15:35:47 <mgoddard> To turn it on its head - are there any reasons why we shouldn't drop centos 7 support in master?
15:35:58 <mgoddard> 1 - kayobe still uses it :D
15:36:17 <mgoddard> but we are three patches away from not using it
15:37:05 <yoctozepto> drop, drop, drop, drop
15:37:08 <hrw> +2 for drop
15:37:09 <mgoddard> oh, there is also bifrost
15:37:11 <mgoddard> :)
15:37:18 <hrw> +2 for dropping bifrost too
15:37:21 <mgoddard> but I think we can live without it for a time
15:37:21 <hrw> ;D
15:37:29 <mgoddard> I'm working on it
15:38:43 <mgoddard> I'm not hearing many reasons to keep it
15:38:50 <mgoddard> so how do we drop?
15:39:00 <mgoddard> k-a first?
15:39:16 <mgoddard> zuul jobs first?
15:39:24 <hrw> zuul, k-a+k
15:39:25 <mgoddard> does anyone want to do it?
15:39:40 <mgoddard> I'm sure it will be therapeutic
15:40:36 <yoctozepto> first zuul, project order not relevant
15:41:39 <mgoddard> looking for a volunteer...
15:42:32 <yoctozepto> I thought you needed a therapy after today
15:42:54 <yoctozepto> but could be me if you insist ;p
15:42:54 <mgoddard> I'll get the zuul ball rolling
15:43:03 <mgoddard> that at least will be easy
15:43:23 <mgoddard> I have little incentive to remove support given kayobe needs it for now :)
15:44:10 <mgoddard> #agreed Drop CentOS 7 jobs, starting with kayobe
15:44:20 <mgoddard> #topic Discuss: Ansible may run more handlers than originally planned https://bugs.launchpad.net/kolla-ansible/+bug/1863510
15:44:22 <openstack> Launchpad bug 1863510 in kolla-ansible "Ansible may run more handlers than originally planned" [Medium,Triaged]
15:44:36 <yoctozepto> what are our options?
15:44:40 <mgoddard> Thanks for spotting this yoctozepto
15:44:48 <mgoddard> It's not great tbh
15:44:51 <yoctozepto> I don't think we should revert the dry patch
15:45:11 <mgoddard> I'd really rather not
15:45:20 <mgoddard> when did it land? Stein?
15:45:31 <yoctozepto> or Train, not earlier
15:45:35 <yoctozepto> let's check then
15:45:55 <mgoddard> train
15:45:57 <mgoddard> phew
15:46:05 <yoctozepto> https://review.opendev.org/647699
15:46:06 <patchbot> patch 647699 - kolla-ansible - Simplify handler conditionals (MERGED) - 13 patch sets
15:46:07 <yoctozepto> mhm
15:47:03 <yoctozepto> added to bug report
15:47:08 <mgoddard> ditto :)
15:47:36 <yoctozepto> Radosław Piliszek (yoctozepto) wrote 36 seconds ago:
15:47:39 <yoctozepto> Mark Goddard (mgoddard) wrote 34 seconds ago:
15:47:44 <yoctozepto> late by 2 seconds!
15:47:59 <mgoddard> it's quite nasty, as it affects all our config file generation and container checking tasks
15:48:08 <mgoddard> yoctozepto suggested include_tasks in a loop
15:48:30 <mgoddard> this would be a significant rewrite of every role's config.yml and check-containers.yml
15:48:49 <yoctozepto> as was the one that introduced the bug
15:49:12 <mgoddard> not really a rewrite, it mostly removed lines
15:49:24 <mgoddard> a lot easier to reason about
15:49:36 <mgoddard> (although apparently still flawed)
15:49:59 <mgoddard> there are also performance concerns with include_tasks at scale
15:50:07 <mgoddard> which we haven't fully investigated
15:50:25 <mgoddard> but committing to it in a big way seems scary
15:50:58 <yoctozepto> hmm
15:52:10 <mgoddard> there is another sort of related issue, I wonder if we could fix both
15:52:23 <mgoddard> if I run kolla-ansible genconfig, then kolla-ansible deploy
15:52:37 <mgoddard> if my config changed but container did not, currently the container is not restarted
15:52:44 <mgoddard> I was thinking about how to address this
15:53:04 <yoctozepto> true
15:53:05 <mgoddard> what if we touched a file in /etc/kolla/<container>
15:53:14 <yoctozepto> we should address both at once
15:53:26 <yoctozepto> otherwise it gets extra work for nothing
15:53:37 <mgoddard> then we have tasks/handlers that remove said file, and trigger a restart if it existed
15:54:46 <mgoddard> just a thought
15:55:49 <mgoddard> another option is kolla_set_configs --check
15:56:07 <mgoddard> checks for changes in config files
15:56:45 <mgoddard> yet another consideration is podman
15:57:08 <mgoddard> if we were to be forced to switch at some point, would we need to change our model?
15:57:15 <hrw> sorry guys, I got lost. can patches fixing the current CI issues get RP+2?
15:57:17 <mgoddard> that would prevent backporting
15:57:18 <yoctozepto> good thinking
15:57:40 <mgoddard> hrw: will do when we have working patches
15:57:40 <yoctozepto> hrw: mgoddard needs to merge them
15:57:55 <hrw> ok
15:58:08 <yoctozepto> mgoddard: just to remember that loops with notify will not save us
15:58:14 <hrw> I had 2-3 other discussion during meeting ;(
15:58:21 <yoctozepto> as they are b0rken
15:59:26 <mgoddard> yoctozepto: true, but if there is something simple we can do once for each container it would help
15:59:57 <yoctozepto> that's the one thing we should not rely on, the rest is a mystery :D
16:00:08 <yoctozepto> so to sum up
16:00:15 <yoctozepto> let's add to bug report
16:00:20 <yoctozepto> what we should remember when fixing
16:00:33 <mgoddard> anyway, I don't think it's critical, but it is quite an important problem we should solve
16:00:50 <mgoddard> ok. who is updating bug report?
16:00:54 <mgoddard> wouldn't want to dup
16:02:00 <yoctozepto> I will
16:02:03 <mgoddard> thanks
16:02:07 <yoctozepto> let's wrap it up now
16:02:10 <mgoddard> and that brings us to the end
16:02:14 <mgoddard> thanks all
16:02:19 <mgoddard> #endmeeting