15:02:48 <mgoddard> #startmeeting kolla
15:02:48 <opendevmeet> Meeting started Wed Jun  2 15:02:48 2021 UTC and is due to finish in 60 minutes.  The chair is mgoddard. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:02:49 <opendevmeet> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
15:02:52 <opendevmeet> The meeting name has been set to 'kolla'
15:03:03 <mgoddard> #topic rollcall
15:03:07 <yoctozepto> \o/
15:03:10 <mgoddard> \oftc
15:03:23 <wuchunyang> \o/
15:03:24 <yoctozepto> o/ o/ o/ freenode
15:04:08 <mgoddard> #topic agenda
15:04:18 <mgoddard> * Roll-call
15:04:20 <mgoddard> * Agenda
15:04:22 <mgoddard> * Announcements
15:04:24 <mgoddard> ** #openstack-kolla now lives on OFTC
15:04:26 <mgoddard> * Review action items from the last meeting
15:04:28 <mgoddard> * CI status
15:04:30 <mgoddard> * Wallaby release planning
15:04:32 <mgoddard> ** Debian bullseye
15:04:34 <mgoddard> ** chrony
15:04:36 <mgoddard> * Xena cycle planning
15:04:38 <mgoddard> ** master branch life cycle https://etherpad.opendev.org/p/kolla-release-process-draft
15:04:40 <mgoddard> * Documentation improvements http://lists.openstack.org/pipermail/openstack-discuss/2021-May/022478.html
15:04:42 <mgoddard> * RabbitMQ upgrade issues https://bugs.launchpad.net/kolla-ansible/+bug/1930293
15:04:44 <mgoddard> * Open discussion
15:04:45 <opendevmeet> Launchpad bug 1930293 in kolla-ansible xena "multinode rabbitmq failing upgrades" [High,Triaged] - Assigned to Radosław Piliszek (yoctozepto)
15:04:46 <mgoddard> #topic announcements
15:04:48 <mgoddard> #info #openstack-kolla now lives on OFTC
15:05:01 <mgoddard> Any others?
15:05:06 <yoctozepto> \o/ \o/ \o/
15:05:14 <yoctozepto> nothing else
15:05:40 <mgoddard> #topic Review action items from the last meeting
15:05:43 <mgoddard> There were none
15:05:48 <mgoddard> #topic CI status
15:05:54 <yoctozepto> there is one we missed last time
15:05:57 <yoctozepto> to redo
15:06:09 <yoctozepto> (mgoddard to write something to the ml)
15:06:18 <yoctozepto> but I can forgive you ;p
15:06:22 <mgoddard> #topic mgoddard email openstack-discuss about quay.io credentials
15:06:29 <yoctozepto> you mean
15:06:31 <yoctozepto> #undo
15:06:34 <mgoddard> yes
15:06:35 <yoctozepto> #action ...
15:06:38 <mgoddard> #action mgoddard email openstack-discuss about quay.io credentials
15:07:03 <mgoddard> #topic CI status
15:07:27 <mgoddard> Kolla seems ok?
15:08:01 <mgoddard> hopefully ovmf-efi issue will get a fix in stream soon
15:08:18 <mgoddard> k-a and kayobe also good afaict
15:08:39 <mgoddard> #topic Wallaby release planning / Debian bullseye
15:09:26 <mgoddard> I think we're looking pretty good here now
15:09:48 <mgoddard> https://review.opendev.org/q/topic:%22debian-bullseye-stable%252Fwallaby%22+(status:open%20OR%20status:merged)
15:09:58 <mgoddard> need approvals on those
15:10:25 <mgoddard> There's this one: https://review.opendev.org/c/openstack/kolla-ansible/+/792583
15:10:34 <mgoddard> (bump min docker version)
15:10:47 <yoctozepto> yeah, I would see that in
15:11:10 <yoctozepto> we could really even backport it to victoria and ussuri
15:11:29 <yoctozepto> though probably "whatever" :-)
15:11:30 <mgoddard> backwards compat?
15:12:00 <mgoddard> there is one potential hole
15:12:26 <mgoddard> what happens if I run docker<v1.41 with cgroups v2?
15:12:37 <mgoddard> I guess it fails to deploy libvirt
15:12:45 <yoctozepto> mgoddard: the first container fails
15:12:47 <mgoddard> but should that be a precheck?
15:12:54 <mgoddard> the first?
15:13:08 <yoctozepto> mgoddard: yeah, I meant docker just fails everything
15:13:35 <yoctozepto> you can't miss it
15:13:49 <mgoddard> ah right
15:13:52 <mgoddard> k
15:14:11 <mgoddard> then I think we have all we need for bullseye
15:14:27 <wuchunyang> victoria seems doesn't support Bullseye in our support matrix.
15:14:48 <mgoddard> we are just adding support for bullseye in wallaby
15:14:55 <yoctozepto> wuchunyang: yeah, it does not; we were discussing https://review.opendev.org/c/openstack/kolla-ansible/+/792583
15:15:08 <yoctozepto> regarding that backporting
15:15:14 <yoctozepto> but it's probably not worth it
15:15:20 <yoctozepto> everybody probably just has it
15:15:25 <mgoddard> #topic Wallaby release planning / chrony
15:15:27 <yoctozepto> at the almost latest
15:15:58 <mgoddard> https://review.opendev.org/c/openstack/kolla-ansible/+/792119
15:16:25 <yoctozepto> yeah, I will re-review surely
15:16:37 <mgoddard> patch to clean up chrony
15:18:03 <mgoddard> similar patch for kayobe: https://review.opendev.org/c/openstack/kayobe/+/792243
15:18:45 <mgoddard> #topic master branch life cycle https://etherpad.opendev.org/p/kolla-release-process-draft
15:19:02 <mgoddard> I'm not sure we'll ever have quorum for this
15:19:22 <mgoddard> #link https://etherpad.opendev.org/p/kolla-release-process-draft
15:19:42 <mgoddard> are there any kinks that need ironing out?
15:22:18 <yoctozepto> we are just not disciplined enough
15:22:22 <yoctozepto> to follow it ;d
15:23:30 <mgoddard> probably not
15:23:55 <mgoddard> I do worry that these extra steps will get forgotten
15:24:04 <yoctozepto> me too
15:24:15 <yoctozepto> I mean, I don't see anything wrong in the plan
15:24:19 <yoctozepto> just that we will not stick to it
15:26:18 <mgoddard> this is R-18
15:26:25 <mgoddard> so next week we should
15:26:27 <mgoddard> Switch source images to use development branches
15:26:29 <mgoddard> Set previous_release variables to the previous release
15:26:47 <yoctozepto> yeah, let's try that
15:26:58 <yoctozepto> and be more coordinated the next time
15:27:15 <yoctozepto> we are getting better and better at it anyhow
15:27:41 <mgoddard> we probably need to reference the schedule each week
15:28:02 <mgoddard> which week is it, what's up next etc.
15:28:34 <priteau> mgoddard: you can add your project milestones to the release calendar
15:28:43 <mgoddard> that's true
15:28:50 <priteau> should help with making them less forgattable
15:30:23 <yoctozepto> and we can have this point in our agenda
15:30:28 <yoctozepto> we have weekly meetings ;p
15:32:03 <mgoddard> the source/binary difference is a bit annoying
15:32:26 <yoctozepto> do you suggest to postpone both till then?
15:32:33 <yoctozepto> sounds fair
15:32:36 <yoctozepto> for our stability ;d
15:33:14 <mgoddard> special cases just add to the congnitive load
15:33:27 <mgoddard> it would be nicer to be able to say
15:33:40 <mgoddard> 'today all images use the previous release'
15:33:48 <mgoddard> 'tomorrow they will all be based on master'
15:34:37 <mgoddard> is the worry that fixes don't get into binary repos?
15:34:45 <mgoddard> (...quickly enough)
15:36:37 <yoctozepto> they are always slower
15:36:50 <yoctozepto> but I think we wait because before there are just no repos to use, no?
15:37:01 <yoctozepto> anyways, binary is n-v
15:37:07 <yoctozepto> I would not care too much
15:37:42 <mgoddard> UCA should exist pretty early in the cycle
15:37:52 <mgoddard> we just usually forget to update it
15:38:05 <mgoddard> RDO master is always available IIUC
15:38:31 <mgoddard> debian isn't even available at openstack release
15:39:07 <mgoddard> alternatively we could just drop binary jobs
15:39:35 <mgoddard> CI should be green
15:40:51 <yoctozepto> we can drop them any time, just tell the word
15:41:03 <yoctozepto> say* the word
15:41:07 <yoctozepto> tell the world
15:41:19 * yoctozepto mixing words
15:44:13 <mgoddard> well I guess we should just give it a try
15:44:16 <mgoddard> and document it
15:44:47 <yoctozepto> agreed
15:45:04 <mgoddard> #action mgoddard document new release process
15:45:13 <yoctozepto> sadly, nobody else is discussing that
15:45:28 <mgoddard> #action start following new release process
15:45:45 <mgoddard> #topic Documentation improvements http://lists.openstack.org/pipermail/openstack-discuss/2021-May/022478.html
15:46:22 <mgoddard> did anyone else read through the recommendations?
15:46:53 <yoctozepto> I find it ok
15:47:04 <yoctozepto> want to see it in docs first
15:47:20 <yoctozepto> I'm glad users have their own proposals
15:47:22 <mgoddard> Klemen responded to the call for contributors and I'd like to make sure we're not blocking their help
15:47:26 <yoctozepto> the docs should be for users
15:48:16 <mgoddard> we could advise on how to break it up
15:48:36 <mgoddard> some less contentious/challenging changes first
15:49:10 <mgoddard> what about the admin/user split
15:49:26 <mgoddard> I think we all agree to put everything under one, but which?
15:49:33 <mgoddard> one of those or something different?
15:49:42 <mgoddard> #link https://etherpad.opendev.org/p/kollakall
15:51:14 <mgoddard> maybe everything under admin?
15:51:45 <yoctozepto> our users are admins/operators
15:51:47 <mgoddard> given that our users are admins
15:51:57 <yoctozepto> /deployers
15:52:21 <yoctozepto> well, they could deploy and not administer
15:52:26 <mgoddard> yes
15:52:27 <yoctozepto> yet be our users
15:52:38 <yoctozepto> everything under user then perhaps?
15:52:49 <mgoddard> sure, I don't mind either way
15:52:55 <yoctozepto> and start with "who is the kolla user"
15:53:25 <mgoddard> so first change could be a simple move of existing admin sections under user, including redirects
15:53:58 <yoctozepto> ++
15:54:01 <yoctozepto> me gusta
15:54:24 <mgoddard> maybe some simple reordering of pages
15:55:15 <mgoddard> then work on the early intro/architecture pages
15:56:19 <mgoddard> ok, I'll follow up on the ML
15:56:31 <mgoddard> #action mgoddard follow up about documentation improvements
15:56:34 <mgoddard> #topic RabbitMQ upgrade issues https://bugs.launchpad.net/kolla-ansible/+bug/1930293
15:56:36 <opendevmeet> Launchpad bug 1930293 in kolla-ansible xena "multinode rabbitmq failing upgrades" [High,Triaged] - Assigned to Radosław Piliszek (yoctozepto)
15:56:47 <mgoddard> I hit this a few times recently
15:56:57 <mgoddard> do you know what's going on?
15:57:01 <yoctozepto> glad, /me fixing
15:57:08 <yoctozepto> yeah, everything in the bug report
15:57:10 <yoctozepto> a-z
15:57:18 <yoctozepto> in summary, we got this other fix
15:57:25 <yoctozepto> that introduced serial restarts
15:57:41 <yoctozepto> and this conflicts with how ansible sees order and our stop logic
15:57:58 <yoctozepto> however, we may just want rolling upgrades now
15:58:01 <yoctozepto> they seem to work
15:58:07 <yoctozepto> https://review.opendev.org/c/openstack/kolla-ansible/+/794026
15:58:34 <yoctozepto> v, w and x should be happy with this solution
15:58:44 <yoctozepto> however, we got the original fix back to train
15:58:57 <yoctozepto> and stein->train, train->ussuri might not be as happy
15:59:00 <yoctozepto> to be established
15:59:18 <yoctozepto> but I will recommend to apply the above fix to vwx
15:59:33 <mgoddard> https://www.rabbitmq.com/upgrade.html#rolling-upgrades
15:59:42 <yoctozepto> yeah, based on that
16:00:05 <yoctozepto> they scare me we could be failing this on train->ussuri for we  have 3.7 in train
16:01:17 <mgoddard> and we're out of time
16:01:22 <mgoddard> thanks everyone
16:01:25 <yoctozepto> thanks mgoddard
16:01:29 <mgoddard> #endmeeting