15:01:04 <mnasiadka> #startmeeting kolla 15:01:04 <opendevmeet> Meeting started Wed Nov 24 15:01:04 2021 UTC and is due to finish in 60 minutes. The chair is mnasiadka. Information about MeetBot at http://wiki.debian.org/MeetBot. 15:01:04 <opendevmeet> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 15:01:04 <opendevmeet> The meeting name has been set to 'kolla' 15:01:18 <mnasiadka> #topic rollcall 15:01:59 <frickler> o/ 15:02:03 <adrian-a> \o 15:02:50 <mgoddard> \o 15:03:29 <mgoddard> mnasiadka: where is that ping list from? 15:03:45 <mnasiadka> mgoddard: from the wiki, we still haven't merged the docs change I think 15:03:58 <mnasiadka> o/ 15:05:45 <headphoneJames> o/ 15:06:18 <adrian-a> https://wiki.openstack.org/wiki/Meetings/Kolla#10_minute_warning 15:06:41 <mnasiadka> #topic agenda 15:06:41 <mnasiadka> * Review action items from the last meeting 15:06:41 <mnasiadka> * CI status 15:06:41 <mnasiadka> * Release tasks 15:06:41 <mnasiadka> * Yoga cycle planning 15:06:41 <mnasiadka> * Bootstrap servers in Kayobe (mgoddard) 15:06:41 <mnasiadka> * Open discussion 15:06:49 <mnasiadka> #topic Review action items from the last meeting 15:07:03 <mnasiadka> mnasiadka to triage security bugs and update them with resolution plan (if needed) 15:07:03 <mnasiadka> yoctozepto hide properly init-runonce 15:07:04 <mnasiadka> not forget to go through backports for stable branches (L248 on Whiteboard) and do stable releases afterwards. 15:07:04 <mnasiadka> mnasiadka to EOL rocky and stein for both kolla and kolla-ansible 15:07:11 <mnasiadka> I didn't triage sec bugs 15:07:18 <mnasiadka> I have raised the EOL change and sent a mail to ML 15:07:36 <mnasiadka> yoctozepto is not here, but I don't think he did his 15:08:00 <mnasiadka> Has anybody gone through backports? I guess not... 15:09:11 <mgoddard> not I 15:09:20 <mnasiadka> Then not 15:09:25 <mnasiadka> #action mnasiadka to triage security bugs and update them with resolution plan (if needed) 15:09:30 <mnasiadka> #action yoctozepto hide properly init-runonce 15:09:39 <mnasiadka> #action not forget to go through backports for stable branches (L248 on Whiteboard) and do stable releases afterwards. 15:09:45 <mnasiadka> #topic CI Status 15:10:03 <mnasiadka> We've had a fail on publish jobs for centos in Ussuri (monasca-grafana failed) 15:11:11 <mnasiadka> hrw was looking at rebuilding it, but I think it suceeded on his end 15:12:08 <mnasiadka> I'll check as well, and if it works, then it was just a one off... (and Ussuri is EM anyway) 15:12:36 <mnasiadka> CI Status on the whiteboard looks green as grass 15:13:04 <mnasiadka> #topic Release tasks 15:13:29 <mnasiadka> it's R-18 now 15:14:05 <mnasiadka> R-17 was Switch source images to current release 15:14:20 <mnasiadka> mgoddard: are those changes merged? I remember you raised those? 15:14:44 <mgoddard> R-17 is next week :) 15:15:06 <mgoddard> they didn't get merged IIRC 15:15:52 <mnasiadka> Ah, right 15:16:01 <mnasiadka> Can we get them marked as priority changes? 15:16:18 <mgoddard> We may as well wait for next week now 15:17:22 <mnasiadka> Sure 15:17:42 <mnasiadka> So let's move on 15:17:49 <mnasiadka> #topic Yoga cycle planning 15:18:37 <mnasiadka> I've sent out the email regarding Kolla plans for next cycle, let's wait a bit more for replies - I know Fl1nt promised one (and some others promised to send replies) 15:19:04 <mnasiadka> Is there any particular feature we'd like to discuss today? 15:19:46 <mgoddard> How is the yoga feature list looking? 15:21:17 <mnasiadka> I started tidying it up and will populate it in the whiteboard, so that's a valid question mgoddard ;-) 15:22:24 <mgoddard> yeah, would be nice to have something to iterate on 15:22:51 <mnasiadka> Ok, I'll make that my priority for this week. 15:22:57 <mgoddard> should we formalise the process of updating/maintaining versions? 15:23:34 <adrian-a> I assume this can be dropped as CentOS will be deprecated: https://bugs.launchpad.net/kolla/+bug/1946801 15:24:06 <mnasiadka> mgoddard: do you mean stable minor releases? 15:24:37 <mgoddard> I think we don't have a very standard way of deciding when we should update things like OS distros, core component versions (mariadb, rabbitmq) and tooling (ansible) 15:24:39 <mnasiadka> adrian-a: no, that is for both binary and source - but we may fix it by not using RDO packages. 15:25:06 <adrian-a> ok 15:25:17 <mgoddard> adrian-a: let's not make any assumptions until we have a final decision on the future of images 15:25:45 <mnasiadka> mgoddard: that's correct, we could think of having some rules 15:25:52 <hrw> mnasiadka: monasca-grafana failed in ussuri, worked in victoria 15:26:28 <mgoddard> I suppose rules might help, e.g. can't update things after R-X 15:26:28 <mnasiadka> Especially after deprecating/dropping binary - we would not be so tied to upgrading OS distros as we currently are with CentOS 15:26:52 <mnasiadka> true 15:27:18 <mgoddard> I was thinking more about encouraging conversations at certain points in a release cycle about what we might need to update 15:27:50 * hrw off, sorry 15:27:52 <yoctozepto> o/ (sorry, I got lost in some analysis and forgot the meeting) 15:28:05 <yoctozepto> (oh my, two sorries at once) 15:28:54 <mnasiadka> Ok then, one thing is updating Ansible - this can be more or less predicted even at PTG level 15:29:00 <mgoddard> hrw tags in yoctozepto 15:29:39 <mnasiadka> Probably OS distro upgrade is similar 15:29:50 <mgoddard> hopefully, yes 15:29:52 <yoctozepto> mgoddard: :D 15:30:28 <yoctozepto> ok, so we have upgrade to mariadb 10.6 as we planned 15:30:34 <yoctozepto> but the change just landed randomly 15:30:39 <yoctozepto> upgraded* 15:30:45 <mgoddard> right, that's what I'm saying 15:30:47 <mnasiadka> And then new RabbitMQ/MariaDB/infrastructure_software releases should not be done late in the cycle, often OS distro upgrade dictates those versions 15:31:22 <mnasiadka> And especially MariaDB/RabbitMQ upgrades should be done early in the cycle to see what issues will we get in CI for the next couple of months 15:31:50 <mgoddard> so maybe we need a standard PTG topic for this, and then a release process point mid-cycle to discuss again 15:31:54 <mnasiadka> But RabbitMQ and MariaDB are now from their own repositories, so we have more control over them. 15:32:05 <yoctozepto> yeah 15:32:12 <yoctozepto> is rabbitmq upgradable btw? 15:32:18 <mnasiadka> You ask me :) 15:32:19 <yoctozepto> I think they have been on 3.8 for a long time 15:32:34 <yoctozepto> yeah, there seems to be 3.9 already 15:32:36 <adrian-a> we have a rule in another project where we update 3rd parties as first task after a stable release (or as early as possible), this way we have a full cycle to discover issues 15:32:39 <yoctozepto> do we want to touch it? 15:32:52 <mnasiadka> Do we need some part of the Kolla contributor docs with standard topics that should be discussed over PTG and then revisited in mid-cycle? 15:33:32 <yoctozepto> we could use that 15:33:40 <yoctozepto> (re: docs) 15:34:33 <mgoddard> +1 15:34:50 <mnasiadka> Ok, any volunteer to start the docs change? 15:35:38 <yoctozepto> now that's a harder question 15:36:22 <mnasiadka> Well, I need to update the weekly meetings docs change, so I can start that one 15:36:53 <mnasiadka> #action mnasiadka post a patch for docs - standard topics that should be discussed over PTG and then revisited in mid-cycle 15:36:56 * yoctozepto cheering mnasiadka on 15:38:50 <mnasiadka> While we're at tooling - ansible-core 2.11 is out, Ansible 5 (the collection of collections and so on) will be out somewhere mid December - are we bumping this cycle? 15:39:24 <mnasiadka> ups, I mean ansible-core 2.12 15:39:32 <yoctozepto> if it works with core 2.12, then I'm all in 15:39:40 <yoctozepto> or otherwise 15:39:45 <mnasiadka> Well, Python 3.8 is a hard requirement ;-) 15:39:55 <yoctozepto> if we can keep the compats sane 15:39:57 <yoctozepto> ah well 15:40:02 <mgoddard> ah, difficult for centos 8 15:40:15 <yoctozepto> for sure we don't want to make it a minimum 15:40:22 <mnasiadka> Yes, even for Stream 15:40:28 <mnasiadka> But we could set it as maximum 15:40:29 <yoctozepto> but let's test it 15:40:32 <yoctozepto> indeed 15:41:19 <mnasiadka> Ok, I'll add it to the list of priorities for Yoga 15:41:38 <mnasiadka> #action mnasiadka Add ansible-core 2.12 to the list of Yoga priorities 15:41:52 <mnasiadka> ok, any other Yoga topics? 15:42:00 <yoctozepto> yeah, rabbitmq 15:42:05 <yoctozepto> do we want to bump it since we can? 15:42:15 <mgoddard> how new is it? 15:42:36 <yoctozepto> 3.9.0 23 July 2021 15:42:38 <mnasiadka> 3.8 End of General Support is 31 January 2022 15:42:43 <mnasiadka> so do we have any option not to? 15:42:44 <yoctozepto> 3.9.10 20 November 2021 15:42:50 <mgoddard> ok, let's go 15:43:02 <mgoddard> it can't get much worse :D 15:43:10 <mnasiadka> #action mnasiadka Add rabbitmq 3.9 to the list of Yoga priorities 15:43:11 <mnasiadka> ;) 15:43:17 <mnasiadka> anything else? 15:43:27 <yoctozepto> 1 General Support means patch releases that are produced regularly based on feedback from all users. 15:43:27 <yoctozepto> 2 Extended Support means security patches and high-severity issues reported by users with a commercial license. 15:43:37 <yoctozepto> just a note it's not THAT BAD ;-) 15:43:50 <mnasiadka> I don't have a commercial license ;-) 15:43:52 <yoctozepto> nothing else from me 15:44:13 <yoctozepto> mnasiadka: yeah, but utterly broken releases will still get fixed as well as sec issues 15:44:58 <yoctozepto> anyhow, let's bump since it's sensible 15:45:00 <mnasiadka> yoctozepto: I don't want to count how many times did I do full RabbitMQ wipe to get it back to working state, and couldn't find a bug that caused that ;-) 15:45:18 <mnasiadka> #topic Bootstrap servers in Kayobe (mgoddard) 15:45:20 <yoctozepto> mnasiadka: precisely, so what are you worrying about even? :D 15:46:05 <mnasiadka> mgoddard: stage is yours 15:46:26 <mgoddard> thank you 15:47:04 <mgoddard> tl;dr: I'm working on copying the baremetal role from kolla-ansible into kayobe 15:47:22 <mgoddard> some background 15:47:38 <mgoddard> Ansible error handling is tricky 15:47:57 <mgoddard> #link https://github.com/markgoddard/ansible-experiments/tree/master/14-error-handling 15:48:46 <mgoddard> The use case is: allow a 'kayobe overcloud host configure' command to execute to completion in the face of some host failures, or even missing hosts 15:49:28 <mgoddard> The relevant part of the problem here is that kayobe mixes its own playbooks and kolla-ansible bootstrap-servers in that command 15:49:50 <mgoddard> so if any hosts fail during the kayobe playbooks, we don't run bootstrap-servers 15:50:23 <mgoddard> some less important reasons for doing this: 15:50:37 <yoctozepto> ( mgoddard: on the linked github page some "echo $?" are missing return value ) 15:51:12 <mgoddard> it can be a bit clunky/hairy to specify tags/limit to this command, since it mixes kayobe & kolla playbooks 15:51:32 <mgoddard> and some parts of bootstrap-servers are duplicated in kayobe already 15:51:49 <mgoddard> finally, some of this code could use a clean up 15:52:28 <opendevreview> Radosław Piliszek proposed openstack/kolla-ansible master: [WIP] Add Ansible 5 aka core 2.12 support https://review.opendev.org/c/openstack/kolla-ansible/+/819135 15:52:42 * yoctozepto does not use kayobe but mgoddard's monologue is sensible 15:53:05 <mnasiadka> mgoddard: I remember we spoke about moving some tasks to separate role than baremetal (e.g. docker) 15:53:29 <mgoddard> the reason I bring this here today is to find out whether there is interest in sharing this code between kolla-ansible and kayobe 15:53:37 <mnasiadka> But I'm worried about copy'ing - e.g. maintaining it in both places - maybe we should create a ,,Kolla'' Galaxy collection? 15:53:42 <mgoddard> or should I just copypasta 15:53:57 <yoctozepto> mgoddard: copypasta sounds bad 15:54:23 <mgoddard> I think a kolla collection would be ideal, it's just whether we want to create and maintain that thing 15:54:24 <yoctozepto> what is your plan about sharing the code? could we standardise something? 15:54:41 <psuedo> o/ 15:54:52 <mgoddard> it would also be a bit more effort to make the code work in both codebases 15:54:54 <yoctozepto> hmm, but then again; what does a collection bring over being in kolla-ansible? 15:55:24 <yoctozepto> I mean, except for distribution :D 15:55:43 <mgoddard> well, what it brings to k-a is benefitting from me spending some time on it 15:56:05 <mnasiadka> Well, I think we can't just leave it in kolla-ansible repo and distribute it to kayobe in the same time? ;-) 15:56:09 <mgoddard> I probably wouldn't keep the two in sync if each had a copy 15:56:27 <yoctozepto> mnasiadka: does not kayobe install kolla-ansible? 15:56:39 <mnasiadka> yoctozepto: in a kolla-ansible venv 15:56:52 <mnasiadka> (separate from kayobe venv) 15:57:24 <yoctozepto> yeah, but, as I said, what's the difference except for distribution means? 15:57:39 <mgoddard> I see your point, yoctozepto 15:57:59 <mgoddard> it could probably be made to work by fudging the role path 15:58:17 <mgoddard> I would be a bit concerned about changes to the role breaking kayobe 15:59:00 <frickler> could add a cross-project job for that? 15:59:14 <yoctozepto> ^ precisely 15:59:21 <mgoddard> we could, and only trigger it on changes to the role 15:59:32 <yoctozepto> I mean, best to integrate closely 15:59:41 <yoctozepto> collection could be trickier in fact 15:59:51 <mgoddard> what'why the resistance to the collection? 15:59:56 <mnasiadka> tricker for kolla-ansible (we would need to add support for that) 15:59:58 <mgoddard> s/what'// 16:00:01 <mnasiadka> in kayobe that's a standard 16:00:27 <yoctozepto> mgoddard, mnasiadka: can we test with in-flight/depends-on patches and a collection? 16:00:45 <yoctozepto> if yes, then it's no brainer and let's just go collection :-) 16:00:48 <mgoddard> sure, if it's maintained in opendev 16:01:31 <mgoddard> we have talked about using ansible-core and collections for kolla-ansible before 16:01:56 <mnasiadka> yup 16:02:15 <mnasiadka> so a collection it is? 16:02:19 <yoctozepto> ok, then it all makes sense to me 16:02:25 <yoctozepto> yeah, seems so 16:02:51 <opendevreview> Doug Szumski proposed openstack/kolla-ansible master: Support copying static Vendordata file into Nova API container https://review.opendev.org/c/openstack/kolla-ansible/+/819140 16:03:04 <mgoddard> #link https://review.opendev.org/c/openstack/kayobe/+/818290 16:03:17 <mgoddard> that's the initial PoC 16:04:15 <mnasiadka> ok then 16:04:42 <mnasiadka> mgoddard: will you handle the addition repo creation and creating a collection? 16:05:11 <mgoddard> I will 16:05:53 <mnasiadka> Ok then, seems like another priority for Yoga, initiated on Kayobe side, but affecting Kolla-Ansible 16:06:44 <mnasiadka> And I think we went a bit over time. 16:07:00 <mnasiadka> Thanks for the meeting :) 16:07:03 <mnasiadka> #stopmeeting 16:07:07 <mnasiadka> #endmeeting