13:00:30 <mnasiadka> #startmeeting kolla 13:00:30 <opendevmeet> Meeting started Wed Sep 27 13:00:30 2023 UTC and is due to finish in 60 minutes. The chair is mnasiadka. Information about MeetBot at http://wiki.debian.org/MeetBot. 13:00:30 <opendevmeet> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 13:00:30 <opendevmeet> The meeting name has been set to 'kolla' 13:00:32 <mnasiadka> #topic rollcall 13:00:32 <mnasiadka> o/ 13:00:49 <jangutter> \o 13:00:54 <bbezak> o/ 13:01:07 <jsuazo> \o 13:01:32 <frickler> \o 13:01:39 <SvenKieske> o/ 13:02:37 <kevko> \o/ 13:04:17 <mnasiadka> #topic agenda 13:04:17 <mnasiadka> * CI status 13:04:17 <mnasiadka> * Release tasks 13:04:17 <mnasiadka> * Regular stable releases (first meeting in a month) 13:04:17 <mnasiadka> * Current cycle planning 13:04:19 <mnasiadka> * Additional agenda (from whiteboard) 13:04:19 <mnasiadka> * Open discussion 13:04:21 <mnasiadka> #topic CI status 13:04:34 <mnasiadka> So, Kolla CI is red, because RabbitMQ went over their CloudSmith credits 13:04:50 <mnasiadka> there's a patch to move to a mirror of CloudSmith, but aarch64 is not there 13:05:09 <mnasiadka> I raised a question if RMQ folks could mirror aarch64 as well 13:05:11 <mnasiadka> #link https://github.com/rabbitmq/rabbitmq-server/discussions/9553 13:05:37 <opendevreview> Juan Pablo Suazo proposed openstack/kolla-ansible master: Deploy Glance and Cinder Backup with S3 backend support https://review.opendev.org/c/openstack/kolla-ansible/+/844614 13:06:11 <mnasiadka> I haven't noticed other problematic failures (apart the Rocky mirror availability that has been solved now) 13:06:21 <mnasiadka> #topic Release tasks 13:06:30 <mnasiadka> I still didn't have time to raise the cycle highlights patch 13:06:41 <mnasiadka> I'll do so today... 13:07:41 <mnasiadka> Once all ,,normal'' projects release soon - we should get RDO packages and can think about a list of things to do before we branch 13:07:57 <mnasiadka> #topic Current cycle planning 13:08:04 <mnasiadka> So, we merged Kolla/Podman 13:08:28 <mnasiadka> Kolla-Ansible/Podman needs reviews - I did have a look on the a-c-k patch, and probably we need a discussion 13:08:36 <mnasiadka> currently we set virtualenv: None 13:08:48 <mnasiadka> and do when: virtualenv is not None 13:09:02 <mnasiadka> and it seems some contributors believe it's an antipattern or bad practice 13:09:15 <mnasiadka> and we should not define variable defaults 13:09:22 <mnasiadka> and use when: virtualenv is not defined 13:09:28 <kevko> which defaults ? 13:09:32 <mnasiadka> role defaults 13:09:32 <mnasiadka> :) 13:09:51 <mnasiadka> we have historically set role defaults to empty variables, sometimes to None and lived like that 13:10:16 <mnasiadka> #link https://review.opendev.org/c/openstack/ansible-collection-kolla/+/892990 13:10:18 <SvenKieske> mhm, I have to think about this again 13:10:25 <mmalchuk> o/ 13:10:28 <kevko> i think defaults should be defaults and host_vars and group_vars or role vars can override ... 13:10:32 <mnasiadka> and it seems podman feature authors have been influenced by this 13:10:53 <mnasiadka> I would prefer we stick to the current ,,scheme'' and discuss such things on PTG 13:11:51 <mnasiadka> Maybe it could be an extension of ,,Zens of Kolla'' including some custom ansible-lint rules or something similar 13:12:08 <SvenKieske> well podman feature authors got afaik told not to do it a certain way and then iirc pointed out that we do this stuff inconsistent? not sure anymore though.. 13:12:10 <mnasiadka> Just not near the cycle end :) 13:12:26 <SvenKieske> didn't we already agree to discuss this during PTG? 13:13:08 <mnasiadka> Yeah, I think so, just wanted to confirm :) 13:13:26 <mnasiadka> but for example here: https://review.opendev.org/c/openstack/ansible-collection-kolla/+/852240/25/roles/podman/defaults/main.yml#20 13:13:50 <mnasiadka> I see commented out variables in role defaults 13:14:24 <mnasiadka> I'll do some testing of that role and rework it so we keep to the current ,,scheme'' of empty but set variables in defaults :) 13:14:45 <mnasiadka> kevko: mentioned we need python3-podman package in Debian (or fail when installing podman-py outside of venv) 13:15:03 <mnasiadka> Probably for now we should fail, and once the package is in Debian - we can do the same what we do for docker 13:15:10 <mnasiadka> kevko: is that fine? 13:15:15 <kevko> yep 13:15:24 <mnasiadka> great 13:15:38 <mnasiadka> Let's Encrypt - I can tackle that one once podman is in 13:15:46 <SvenKieske> just pointing out the obvious: we could also force a venv for podman, no? it's a new feature? 13:16:24 <mnasiadka> SvenKieske: I think that implies we would force a venv for kolla-ansible and we would need to glue those two things together 13:16:34 <mnasiadka> up until now we haven't forced that on users 13:16:38 <SvenKieske> I'm also fine with the current solution, just wanted to mention that in case it was missed. 13:16:45 <mnasiadka> We could discuss that at the PTG 13:16:49 <SvenKieske> ok 13:17:04 <mnasiadka> bbezak: unless you have cycles to have a look in Let's Encrypt in coming days? 13:17:45 <kevko> i hope LE is written nice now :) 13:18:02 <mnasiadka> kevko: just looking for a thorough review :) 13:18:17 <kevko> i know 13:18:33 <bbezak> sorry, not in 2-3 weeks (going on vacation) 13:19:12 <kevko> regarding python-docker and python-podman ...this module is needed on dest hosts ... so does it mean that you want to install venv on dest hosts and change PYTHONPATH to include venv ? 13:19:27 <mnasiadka> ok then, LE waiting for Podman to merge ;) 13:20:10 <mnasiadka> kevko: I think let's not get into that now, just fail if Debian and not venv ;) We can discuss options at the PTG if the package does not get into Debian 13:20:52 <mnasiadka> jangutter: how are we with etcd bump? is there a patch to add a precheck that Zun does not work at all now? 13:21:03 <mnasiadka> (and a deprecation notice attached) 13:21:29 <mmalchuk> there was a change to drop Zun 13:21:53 <jangutter> I've sent up one last night (sorry got busy with on-call stuff here) 13:21:54 <mmalchuk> https://review.opendev.org/c/openstack/kolla-ansible/+/896593 13:22:36 <mnasiadka> looks nice, thanks 13:22:50 <mnasiadka> kevko, frickler - mind having a look at ^^ ? 13:23:25 <jangutter> Any notes are welcome, since this is mostly going to be a human-interaction thing, if you can find anything that's confusing or should be simpler, don't be afraid to yell! 13:24:14 <opendevreview> Michal Nasiadka proposed openstack/kolla master: Drop docker-py from requirements.txt https://review.opendev.org/c/openstack/kolla/+/896644 13:24:30 <mnasiadka> ok then 13:24:41 <mnasiadka> what about bumping etcd now? what's the order jangutter ? 13:25:28 <jangutter> Order is the role improvements then the version bump I guess. (doesn't matter much for folks running releases, but it makes it a bit more testable in CI) 13:26:03 <opendevreview> Merged openstack/kolla-ansible master: Add ML2/OVN and ML2/OVS setting checks for neutron https://review.opendev.org/c/openstack/kolla-ansible/+/895143 13:26:04 <jangutter> but, they're technically totally independent. 13:26:08 <mnasiadka> ok then 13:26:24 <SvenKieske> I yelled :) 13:26:34 <jangutter> Thanks! 13:26:43 <mnasiadka> #link https://review.opendev.org/q/topic:update-etcd-v3.4 13:26:45 <SvenKieske> deprecation != disable would be nice to correct that, it's a pet peeve of mine 13:26:46 <mnasiadka> so that one 13:27:34 <opendevreview> Michal Nasiadka proposed openstack/kolla master: Fix build errors missing podman/docker module https://review.opendev.org/c/openstack/kolla/+/896586 13:27:53 <mnasiadka> ok, I think that's it for current cycle, I mean enough things to review for a week ;) 13:28:07 <mnasiadka> #topic Additional agenda (from whiteboard) 13:28:36 <mnasiadka> (jsuazo) cinder & galnce S3 support. Ready for merge (?) 13:28:47 <mmalchuk> almost 13:29:00 <mmalchuk> cleanup the commit message and merge 13:30:07 <jsuazo> Will make the change in about a min, would appreciate the +2 13:30:12 <mnasiadka> yes, jsuazo can you apply mmalchuk's comment? 13:30:35 <mnasiadka> added that to RP+1, hopefully it will get more reviews 13:31:05 <opendevreview> Juan Pablo Suazo proposed openstack/kolla-ansible master: Deploy Glance and Cinder Backup with S3 backend support https://review.opendev.org/c/openstack/kolla-ansible/+/844614 13:31:10 <mnasiadka> (jsuazo) TAAS configurations in kolla 13:31:31 <mnasiadka> ok, that one - I thought we agreed to install TAAS package directly via pip without using additions/sources/whatever? 13:31:46 <mnasiadka> because if we're going to do it like that - we'll need to track versions in sources.py 13:32:15 <opendevreview> Alex Welsh proposed openstack/kayobe master: Add option to skip kolla docker registry login https://review.opendev.org/c/openstack/kayobe/+/896655 13:32:21 <jsuazo> i think the idea was proposed but not settled on 13:33:07 <jsuazo> we could track versions in sources.py as the TaaS repository has Version branches for each openstack version 13:33:23 <mnasiadka> yes, but it also has a version pin in upper-constraints.txt 13:33:32 <mnasiadka> so it should work properly 13:33:47 <mmalchuk> we are talking about master? 13:34:05 <mmalchuk> imho there is no issue in stable branches? 13:34:08 <opendevreview> Rafal Lewandowski proposed openstack/kolla-ansible stable/2023.1: Add ML2/OVN and ML2/OVS setting checks for neutron https://review.opendev.org/c/openstack/kolla-ansible/+/896532 13:35:06 <mnasiadka> well, if we could do it in a simple way - we could backport if anybody is interested - it's just adding a package from pypi 13:36:09 <jsuazo> ok, I will look into changing the installation to be done directly with pip and return to you next week 13:36:29 <mnasiadka> great 13:36:44 <mnasiadka> once that's solved - we can have a look on kolla-ansible side 13:37:49 <jsuazo> ok, thnks! 13:38:39 <mnasiadka> (reviewed the k-a patch now as well) 13:38:49 <mnasiadka> ok then 13:38:52 <mnasiadka> #topic Open discussion 13:38:54 <mnasiadka> Anything? 13:38:56 <kevko> yes 13:39:02 <jsuazo> (glance-cinder proposal with latest changes is live) 13:39:04 <kevko> we have rabbitmq images broken :D 13:39:13 <kevko> erlang/rabbitmq incompatibility 13:39:26 <mnasiadka> yes, we'll solve that when backporting the mirror switch 13:39:27 <mnasiadka> no worries 13:39:48 <kevko> you need to ping packages 13:39:51 <kevko> pin 13:39:55 <mmalchuk> http2 merged with an issue 13:39:59 <mmalchuk> quick fix: https://review.opendev.org/c/openstack/kolla-ansible/+/896609 13:40:40 <kevko> and i have a question , do we have some opinion what to do with rabbitmq images if they are not supported anymore ? 13:40:57 <kevko> (i mean for example for stable branches - if we should fix in stable branch or maintain downstream) 13:41:11 <kevko> mmalchuk: will check 13:42:07 <SvenKieske> you mean no upstream support from rabbitmq or what kind of support? 13:42:12 <mnasiadka> kevko: I had an idea to label rabbitmq container images with a version label and work out a kolla-ansible logic to compare labels and run upgrades if needed (but only where we support rolling ones) 13:42:20 <mnasiadka> I think kevko means EOL versions 13:42:30 <kevko> mnasiadka: yes.. 13:42:51 <SvenKieske> so no upstream support, that's what I meant as well :) 13:42:55 <mnasiadka> rolling upgrades are 3.8+ 13:43:01 <mnasiadka> so my plan should work 13:43:15 <kevko> i noticed that when i was debugging issue on customer side ... i found the incompatibility with erlang ... (we have 25.3 instead 25.2) ...but moreover ...in log i saw "this is not supported anymore" 13:43:19 <SvenKieske> that sounds like a nice idea 13:43:39 <mnasiadka> but then rolling upgrade works only from 3.8 to 3.9, from 3.9 to 3.10 and so on 13:43:53 <SvenKieske> if we put it behind a feature flag|variable "enable_rolling_rabbitmq_upgrades" or something 13:43:56 <mnasiadka> which might mean we would need to do some tags dance 13:44:03 <mnasiadka> (container image tags) 13:44:04 <SvenKieske> some people might want to pin it for some strange reason 13:44:34 <mnasiadka> I'll do some analysis and come up with a proposal on the PTG 13:45:19 <frickler> just EOL old branches faster *scnr* 13:45:35 <mnasiadka> frickler: we're doing that already 13:45:37 <mnasiadka> :) 13:45:55 <mnasiadka> yoga is going to be EM soon, so might be the situation is solved faster ;) 13:45:59 <frickler> well not fast enough, or we wouldn't have this issue with yoga 13:46:58 <mnasiadka> zed is 3.10 and is EOL as well 13:46:59 <SvenKieske> if upgrades wouldn't introduce breaking changes from time to time it wouldn't really matter how fast stuff get's EOL'ed 13:47:23 <mnasiadka> and antelope is 3.11 and will be EOL in 3 months 13:48:51 <frickler> yeah, anyway, I was only be half serious at most 13:48:52 <kevko> EOL - do you think our users are soo fast in upgrading ? :D 13:48:58 <frickler> *being 13:49:11 <SvenKieske> one day devs will EOL faster than they release new versions :D 13:49:56 <kevko> btw - regarding EOL branches .. 13:50:07 <mnasiadka> Ok then, it seems it's a valid topic, at least RabbitMQ is a bit critical in our setups :) 13:50:20 <kevko> what about new policy to merge patchsets for EOL branchces only with one +2 ? 13:50:26 <kevko> is it possible ? 13:50:28 <mnasiadka> EOL branches do not exist 13:50:47 <mnasiadka> EOL = no branch, only wallaby-eol tag, no merging of anything possible 13:50:49 <kevko> ok - stable branches ..not last 13:50:52 <mnasiadka> you mentioned EM branches 13:50:58 <kevko> but release before for example 13:51:00 <kevko> yes 13:51:31 <kevko> i think it's good idea .. wdyt ? 13:52:39 <mnasiadka> We can discuss on the PTG, I have a contrary idea to not maintain EM at all and just do EOL 13:52:47 <frickler> depends on how many reviewers still care about that branch I guess. and would likely better be documented somehow 13:52:56 <SvenKieske> mhm I think cores should decide, because they have to put in the work. I can't judge if it is avoiding sufficient amount of work to be worthwhile. 13:54:07 <mnasiadka> True, but we should be more specific in how do we maintain "unsupported" branches - I think there was a topic going on to rename EM to Unsupported 13:54:09 <frickler> also EM is being replaced by "Unmaintained", but I didn't look at the details yet and how they apply to deployment projects 13:54:17 <mnasiadka> Ah, Unmaintained 13:54:29 <mnasiadka> https://docs.openstack.org/project-team-guide/stable-branches.html#extended-maintenance 13:54:53 <mnasiadka> I'm wondering if there's a knob called "Kolla does not want to do EM/Unmaintained - move EM branch candidates to EOL directly" 13:55:07 <mnasiadka> But if anybody wants to maintain them - sure, we can discuss that on the PTG 13:55:08 <frickler> https://governance.openstack.org/tc/resolutions/20230724-unmaintained-branches.html is the current resolution 13:55:19 <mnasiadka> From my perspective we have problems merging backports in non-EM branches :D 13:55:42 <mnasiadka> Ah right, frickler congrats on getting elected to TC ;-) 13:56:03 <frickler> thx, that was quite some surprise to me 13:56:29 <mnasiadka> well, you put your name in the contest ;) 13:56:54 <mnasiadka> anyway, I'll add the PTG topics we discussed today to the PTG etherpad 13:57:06 <kevko> cg 13:57:13 <SvenKieske> regarding: "move EM branch candidates to EOL directly" afaik we can do that, this was in fact even possible under the old policy, just not done, iirc. 13:57:24 <mnasiadka> and I'd like to discuss the PTG time slots on next meeting - because not a lot of people are voting on the time slots 13:57:34 <mnasiadka> so we just decide on the next meeting about those slots 13:57:51 <SvenKieske> yeah I'm guilty of not voting for some of the time slots, will do.. 13:57:53 <mnasiadka> come with ideas :) 13:58:04 <kevko> link ? :D 13:58:21 <mnasiadka> kevko: do you sometimes read emails? 13:58:37 <kevko> hmm...sometimes :D 13:58:38 <mnasiadka> #link https://lists.openstack.org/pipermail/openstack-discuss/2023-September/035120.html 13:59:16 <SvenKieske> a whole day for kayobe feels excessive ;) 13:59:25 <SvenKieske> just kidding 13:59:52 <kevko> hmm ..i need to try kayobee one day :D 13:59:55 <mnasiadka> https://github.com/rabbitmq/rabbitmq-server/discussions/9553 - I love that RMQ openness - a lot of statements and discussion locked 14:00:15 <mnasiadka> ok, see you next week - thanks for coming :) 14:00:17 <mnasiadka> #endmeeting