15:01:04 <mnasiadka> #startmeeting kolla
15:01:04 <opendevmeet> Meeting started Wed Nov 24 15:01:04 2021 UTC and is due to finish in 60 minutes.  The chair is mnasiadka. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:01:04 <opendevmeet> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
15:01:04 <opendevmeet> The meeting name has been set to 'kolla'
15:01:18 <mnasiadka> #topic rollcall
15:01:59 <frickler> o/
15:02:03 <adrian-a> \o
15:02:50 <mgoddard> \o
15:03:29 <mgoddard> mnasiadka: where is that ping list from?
15:03:45 <mnasiadka> mgoddard: from the wiki, we still haven't merged the docs change I think
15:03:58 <mnasiadka> o/
15:05:45 <headphoneJames> o/
15:06:18 <adrian-a> https://wiki.openstack.org/wiki/Meetings/Kolla#10_minute_warning
15:06:41 <mnasiadka> #topic agenda
15:06:41 <mnasiadka> * Review action items from the last meeting
15:06:41 <mnasiadka> * CI status
15:06:41 <mnasiadka> * Release tasks
15:06:41 <mnasiadka> * Yoga cycle planning
15:06:41 <mnasiadka> * Bootstrap servers in Kayobe (mgoddard)
15:06:41 <mnasiadka> * Open discussion
15:06:49 <mnasiadka> #topic Review action items from the last meeting
15:07:03 <mnasiadka> mnasiadka to triage security bugs and update them with resolution plan (if needed)
15:07:03 <mnasiadka> yoctozepto hide properly init-runonce
15:07:04 <mnasiadka> not forget to go through backports for stable branches (L248 on Whiteboard) and do stable releases afterwards.
15:07:04 <mnasiadka> mnasiadka to EOL rocky and stein for both kolla and kolla-ansible
15:07:11 <mnasiadka> I didn't triage sec bugs
15:07:18 <mnasiadka> I have raised the EOL change and sent a mail to ML
15:07:36 <mnasiadka> yoctozepto is not here, but I don't think he did his
15:08:00 <mnasiadka> Has anybody gone through backports? I guess not...
15:09:11 <mgoddard> not I
15:09:20 <mnasiadka> Then not
15:09:25 <mnasiadka> #action mnasiadka to triage security bugs and update them with resolution plan (if needed)
15:09:30 <mnasiadka> #action yoctozepto hide properly init-runonce
15:09:39 <mnasiadka> #action not forget to go through backports for stable branches (L248 on Whiteboard) and do stable releases afterwards.
15:09:45 <mnasiadka> #topic CI Status
15:10:03 <mnasiadka> We've had a fail on publish jobs for centos in Ussuri (monasca-grafana failed)
15:11:11 <mnasiadka> hrw was looking at rebuilding it, but I think it suceeded on his end
15:12:08 <mnasiadka> I'll check as well, and if it works, then it was just a one off... (and Ussuri is EM anyway)
15:12:36 <mnasiadka> CI Status on the whiteboard looks green as grass
15:13:04 <mnasiadka> #topic Release tasks
15:13:29 <mnasiadka> it's R-18 now
15:14:05 <mnasiadka> R-17 was Switch source images to current release
15:14:20 <mnasiadka> mgoddard: are those changes merged? I remember you raised those?
15:14:44 <mgoddard> R-17 is next week :)
15:15:06 <mgoddard> they didn't get merged IIRC
15:15:52 <mnasiadka> Ah, right
15:16:01 <mnasiadka> Can we get them marked as priority changes?
15:16:18 <mgoddard> We may as well wait for next week now
15:17:22 <mnasiadka> Sure
15:17:42 <mnasiadka> So let's move on
15:17:49 <mnasiadka> #topic Yoga cycle planning
15:18:37 <mnasiadka> I've sent out the email regarding Kolla plans for next cycle, let's wait a bit more for replies - I know Fl1nt promised one (and some others promised to send replies)
15:19:04 <mnasiadka> Is there any particular feature we'd like to discuss today?
15:19:46 <mgoddard> How is the yoga feature list looking?
15:21:17 <mnasiadka> I started tidying it up and will populate it in the whiteboard, so that's a valid question mgoddard ;-)
15:22:24 <mgoddard> yeah, would be nice to have something to iterate on
15:22:51 <mnasiadka> Ok, I'll make that my priority for this week.
15:22:57 <mgoddard> should we formalise the process of updating/maintaining versions?
15:23:34 <adrian-a> I assume this can be dropped as CentOS will be deprecated: https://bugs.launchpad.net/kolla/+bug/1946801
15:24:06 <mnasiadka> mgoddard: do you mean stable minor releases?
15:24:37 <mgoddard> I think we don't have a very standard way of deciding when we should update things like OS distros, core component versions (mariadb, rabbitmq) and tooling (ansible)
15:24:39 <mnasiadka> adrian-a: no, that is for both binary and source - but we may fix it by not using RDO packages.
15:25:06 <adrian-a> ok
15:25:17 <mgoddard> adrian-a: let's not make any assumptions until we have a final decision on the future of images
15:25:45 <mnasiadka> mgoddard: that's correct, we could think of having some rules
15:25:52 <hrw> mnasiadka: monasca-grafana failed in ussuri, worked in victoria
15:26:28 <mgoddard> I suppose rules might help, e.g. can't update things after R-X
15:26:28 <mnasiadka> Especially after deprecating/dropping binary - we would not be so tied to upgrading OS distros as we currently are with CentOS
15:26:52 <mnasiadka> true
15:27:18 <mgoddard> I was thinking more about encouraging conversations at certain points in a release cycle about what we might need to update
15:27:50 * hrw off, sorry
15:27:52 <yoctozepto> o/ (sorry, I got lost in some analysis and forgot the meeting)
15:28:05 <yoctozepto> (oh my, two sorries at once)
15:28:54 <mnasiadka> Ok then, one thing is updating Ansible - this can be more or less predicted even at PTG level
15:29:00 <mgoddard> hrw tags in yoctozepto
15:29:39 <mnasiadka> Probably OS distro upgrade is similar
15:29:50 <mgoddard> hopefully, yes
15:29:52 <yoctozepto> mgoddard: :D
15:30:28 <yoctozepto> ok, so we have upgrade to mariadb 10.6 as we planned
15:30:34 <yoctozepto> but the change just landed randomly
15:30:39 <yoctozepto> upgraded*
15:30:45 <mgoddard> right, that's what I'm saying
15:30:47 <mnasiadka> And then new RabbitMQ/MariaDB/infrastructure_software releases should not be done late in the cycle, often OS distro upgrade dictates those versions
15:31:22 <mnasiadka> And especially MariaDB/RabbitMQ upgrades should be done early in the cycle to see what issues will we get in CI for the next couple of months
15:31:50 <mgoddard> so maybe we need a standard PTG topic for this, and then a release process point mid-cycle to discuss again
15:31:54 <mnasiadka> But RabbitMQ and MariaDB are now from their own repositories, so we have more control over them.
15:32:05 <yoctozepto> yeah
15:32:12 <yoctozepto> is rabbitmq upgradable btw?
15:32:18 <mnasiadka> You ask me :)
15:32:19 <yoctozepto> I think they have been on 3.8 for a long time
15:32:34 <yoctozepto> yeah, there seems to be 3.9 already
15:32:36 <adrian-a> we have a rule in another project where we update 3rd parties as first task after a stable release (or as early as possible), this way we have a full cycle to discover issues
15:32:39 <yoctozepto> do we want to touch it?
15:32:52 <mnasiadka> Do we need some part of the Kolla contributor docs with standard topics that should be discussed over PTG and then revisited in mid-cycle?
15:33:32 <yoctozepto> we could use that
15:33:40 <yoctozepto> (re: docs)
15:34:33 <mgoddard> +1
15:34:50 <mnasiadka> Ok, any volunteer to start the docs change?
15:35:38 <yoctozepto> now that's a harder question
15:36:22 <mnasiadka> Well, I need to update the weekly meetings docs change, so I can start that one
15:36:53 <mnasiadka> #action mnasiadka post a patch for docs - standard topics that should be discussed over PTG and then revisited in mid-cycle
15:36:56 * yoctozepto cheering mnasiadka on
15:38:50 <mnasiadka> While we're at tooling - ansible-core 2.11 is out, Ansible 5 (the collection of collections and so on) will be out somewhere mid December - are we bumping this cycle?
15:39:24 <mnasiadka> ups, I mean ansible-core 2.12
15:39:32 <yoctozepto> if it works with core 2.12, then I'm all in
15:39:40 <yoctozepto> or otherwise
15:39:45 <mnasiadka> Well, Python 3.8 is a hard requirement ;-)
15:39:55 <yoctozepto> if we can keep the compats sane
15:39:57 <yoctozepto> ah well
15:40:02 <mgoddard> ah, difficult for centos 8
15:40:15 <yoctozepto> for sure we don't want to make it a minimum
15:40:22 <mnasiadka> Yes, even for Stream
15:40:28 <mnasiadka> But we could set it as maximum
15:40:29 <yoctozepto> but let's test it
15:40:32 <yoctozepto> indeed
15:41:19 <mnasiadka> Ok, I'll add it to the list of priorities for Yoga
15:41:38 <mnasiadka> #action mnasiadka Add ansible-core 2.12 to the list of Yoga priorities
15:41:52 <mnasiadka> ok, any other Yoga topics?
15:42:00 <yoctozepto> yeah, rabbitmq
15:42:05 <yoctozepto> do we want to bump it since we can?
15:42:15 <mgoddard> how new is it?
15:42:36 <yoctozepto> 3.9.0	23 July 2021
15:42:38 <mnasiadka> 3.8 End of General Support is 31 January 2022
15:42:43 <mnasiadka> so do we have any option not to?
15:42:44 <yoctozepto> 3.9.10	20 November 2021
15:42:50 <mgoddard> ok, let's go
15:43:02 <mgoddard> it can't get much worse :D
15:43:10 <mnasiadka> #action mnasiadka Add rabbitmq 3.9 to the list of Yoga priorities
15:43:11 <mnasiadka> ;)
15:43:17 <mnasiadka> anything else?
15:43:27 <yoctozepto> 1 General Support means patch releases that are produced regularly based on feedback from all users.
15:43:27 <yoctozepto> 2 Extended Support means security patches and high-severity issues reported by users with a commercial license.
15:43:37 <yoctozepto> just a note it's not THAT BAD ;-)
15:43:50 <mnasiadka> I don't have a commercial license ;-)
15:43:52 <yoctozepto> nothing else from me
15:44:13 <yoctozepto> mnasiadka: yeah, but utterly broken releases will still get fixed as well as sec issues
15:44:58 <yoctozepto> anyhow, let's bump since it's sensible
15:45:00 <mnasiadka> yoctozepto: I don't want to count how many times did I do full RabbitMQ wipe to get it back to working state, and couldn't find a bug that caused that ;-)
15:45:18 <mnasiadka> #topic Bootstrap servers in Kayobe (mgoddard)
15:45:20 <yoctozepto> mnasiadka: precisely, so what are you worrying about even? :D
15:46:05 <mnasiadka> mgoddard: stage is yours
15:46:26 <mgoddard> thank you
15:47:04 <mgoddard> tl;dr: I'm working on copying the baremetal role from kolla-ansible into kayobe
15:47:22 <mgoddard> some background
15:47:38 <mgoddard> Ansible error handling is tricky
15:47:57 <mgoddard> #link https://github.com/markgoddard/ansible-experiments/tree/master/14-error-handling
15:48:46 <mgoddard> The use case is: allow a 'kayobe overcloud host configure' command to execute to completion in the face of some host failures, or even missing hosts
15:49:28 <mgoddard> The relevant part of the problem here is that kayobe mixes its own playbooks and kolla-ansible bootstrap-servers in that command
15:49:50 <mgoddard> so if any hosts fail during the kayobe playbooks, we don't run bootstrap-servers
15:50:23 <mgoddard> some less important reasons for doing this:
15:50:37 <yoctozepto> ( mgoddard: on the linked github page some "echo $?" are missing return value )
15:51:12 <mgoddard> it can be a bit clunky/hairy to specify tags/limit to this command, since it mixes kayobe & kolla playbooks
15:51:32 <mgoddard> and some parts of bootstrap-servers are duplicated in kayobe already
15:51:49 <mgoddard> finally, some of this code could use a clean up
15:52:28 <opendevreview> Radosław Piliszek proposed openstack/kolla-ansible master: [WIP] Add Ansible 5 aka core 2.12 support  https://review.opendev.org/c/openstack/kolla-ansible/+/819135
15:52:42 * yoctozepto does not use kayobe but mgoddard's monologue is sensible
15:53:05 <mnasiadka> mgoddard: I remember we spoke about moving some tasks to separate role than baremetal (e.g. docker)
15:53:29 <mgoddard> the reason I bring this here today is to find out whether there is interest in sharing this code between kolla-ansible and kayobe
15:53:37 <mnasiadka> But I'm worried about copy'ing - e.g. maintaining it in both places - maybe we should create a ,,Kolla'' Galaxy collection?
15:53:42 <mgoddard> or should I just copypasta
15:53:57 <yoctozepto> mgoddard: copypasta sounds bad
15:54:23 <mgoddard> I think a kolla collection would be ideal, it's just whether we want to create and maintain that thing
15:54:24 <yoctozepto> what is your plan about sharing the code? could we standardise something?
15:54:41 <psuedo> o/
15:54:52 <mgoddard> it would also be a bit more effort to make the code work in both codebases
15:54:54 <yoctozepto> hmm, but then again; what does a collection bring over being in kolla-ansible?
15:55:24 <yoctozepto> I mean, except for distribution :D
15:55:43 <mgoddard> well, what it brings to k-a is benefitting from me spending some time on it
15:56:05 <mnasiadka> Well, I think we can't just leave it in kolla-ansible repo and distribute it to kayobe in the same time? ;-)
15:56:09 <mgoddard> I probably wouldn't keep the two in sync if each had a copy
15:56:27 <yoctozepto> mnasiadka: does not kayobe install kolla-ansible?
15:56:39 <mnasiadka> yoctozepto: in a kolla-ansible venv
15:56:52 <mnasiadka> (separate from kayobe venv)
15:57:24 <yoctozepto> yeah, but, as I said, what's the difference except for distribution means?
15:57:39 <mgoddard> I see your point, yoctozepto
15:57:59 <mgoddard> it could probably be made to work by fudging the role path
15:58:17 <mgoddard> I would be a bit concerned about changes to the role breaking kayobe
15:59:00 <frickler> could add a cross-project job for that?
15:59:14 <yoctozepto> ^ precisely
15:59:21 <mgoddard> we could, and only trigger it on changes to the role
15:59:32 <yoctozepto> I mean, best to integrate closely
15:59:41 <yoctozepto> collection could be trickier in fact
15:59:51 <mgoddard> what'why the resistance to the collection?
15:59:56 <mnasiadka> tricker for kolla-ansible (we would need to add support for that)
15:59:58 <mgoddard> s/what'//
16:00:01 <mnasiadka> in kayobe that's a standard
16:00:27 <yoctozepto> mgoddard, mnasiadka: can we test with in-flight/depends-on patches and a collection?
16:00:45 <yoctozepto> if yes, then it's no brainer and let's just go collection :-)
16:00:48 <mgoddard> sure, if it's maintained in opendev
16:01:31 <mgoddard> we have talked about using ansible-core and collections for kolla-ansible before
16:01:56 <mnasiadka> yup
16:02:15 <mnasiadka> so a collection it is?
16:02:19 <yoctozepto> ok, then it all makes sense to me
16:02:25 <yoctozepto> yeah, seems so
16:02:51 <opendevreview> Doug Szumski proposed openstack/kolla-ansible master: Support copying static Vendordata file into Nova API container  https://review.opendev.org/c/openstack/kolla-ansible/+/819140
16:03:04 <mgoddard> #link https://review.opendev.org/c/openstack/kayobe/+/818290
16:03:17 <mgoddard> that's the initial PoC
16:04:15 <mnasiadka> ok then
16:04:42 <mnasiadka> mgoddard: will you handle the addition repo creation and creating a collection?
16:05:11 <mgoddard> I will
16:05:53 <mnasiadka> Ok then, seems like another priority for Yoga, initiated on Kayobe side, but affecting Kolla-Ansible
16:06:44 <mnasiadka> And I think we went a bit over time.
16:07:00 <mnasiadka> Thanks for the meeting :)
16:07:03 <mnasiadka> #stopmeeting
16:07:07 <mnasiadka> #endmeeting