13:00:30 <mnasiadka> #startmeeting kolla
13:00:30 <opendevmeet> Meeting started Wed Sep 27 13:00:30 2023 UTC and is due to finish in 60 minutes.  The chair is mnasiadka. Information about MeetBot at http://wiki.debian.org/MeetBot.
13:00:30 <opendevmeet> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
13:00:30 <opendevmeet> The meeting name has been set to 'kolla'
13:00:32 <mnasiadka> #topic rollcall
13:00:32 <mnasiadka> o/
13:00:49 <jangutter> \o
13:00:54 <bbezak> o/
13:01:07 <jsuazo> \o
13:01:32 <frickler> \o
13:01:39 <SvenKieske> o/
13:02:37 <kevko> \o/
13:04:17 <mnasiadka> #topic agenda
13:04:17 <mnasiadka> * CI status
13:04:17 <mnasiadka> * Release tasks
13:04:17 <mnasiadka> * Regular stable releases (first meeting in a month)
13:04:17 <mnasiadka> * Current cycle planning
13:04:19 <mnasiadka> * Additional agenda (from whiteboard)
13:04:19 <mnasiadka> * Open discussion
13:04:21 <mnasiadka> #topic CI status
13:04:34 <mnasiadka> So, Kolla CI is red, because RabbitMQ went over their CloudSmith credits
13:04:50 <mnasiadka> there's a patch to move to a mirror of CloudSmith, but aarch64 is not there
13:05:09 <mnasiadka> I raised a question if RMQ folks could mirror aarch64 as well
13:05:11 <mnasiadka> #link https://github.com/rabbitmq/rabbitmq-server/discussions/9553
13:05:37 <opendevreview> Juan Pablo Suazo proposed openstack/kolla-ansible master: Deploy Glance and Cinder Backup with S3 backend support  https://review.opendev.org/c/openstack/kolla-ansible/+/844614
13:06:11 <mnasiadka> I haven't noticed other problematic failures (apart the Rocky mirror availability that has been solved now)
13:06:21 <mnasiadka> #topic Release tasks
13:06:30 <mnasiadka> I still didn't have time to raise the cycle highlights patch
13:06:41 <mnasiadka> I'll do so today...
13:07:41 <mnasiadka> Once all ,,normal'' projects release soon - we should get RDO packages and can think about a list of things to do before we branch
13:07:57 <mnasiadka> #topic Current cycle planning
13:08:04 <mnasiadka> So, we merged Kolla/Podman
13:08:28 <mnasiadka> Kolla-Ansible/Podman needs reviews - I did have a look on the a-c-k patch, and probably we need a discussion
13:08:36 <mnasiadka> currently we set virtualenv: None
13:08:48 <mnasiadka> and do when: virtualenv is not None
13:09:02 <mnasiadka> and it seems some contributors believe it's an antipattern or bad practice
13:09:15 <mnasiadka> and we should not define variable defaults
13:09:22 <mnasiadka> and use when: virtualenv is not defined
13:09:28 <kevko> which defaults ?
13:09:32 <mnasiadka> role defaults
13:09:32 <mnasiadka> :)
13:09:51 <mnasiadka> we have historically set role defaults to empty variables, sometimes to None and lived like that
13:10:16 <mnasiadka> #link https://review.opendev.org/c/openstack/ansible-collection-kolla/+/892990
13:10:18 <SvenKieske> mhm, I have to think about this again
13:10:25 <mmalchuk> o/
13:10:28 <kevko> i think defaults should be  defaults and host_vars and group_vars or role vars can override ...
13:10:32 <mnasiadka> and it seems podman feature authors have been influenced by this
13:10:53 <mnasiadka> I would prefer we stick to the current ,,scheme'' and discuss such things on PTG
13:11:51 <mnasiadka> Maybe it could be an extension of ,,Zens of Kolla'' including some custom ansible-lint rules or something similar
13:12:08 <SvenKieske> well podman feature authors got afaik told not to do it a certain way and then iirc pointed out that we do this stuff inconsistent? not sure anymore though..
13:12:10 <mnasiadka> Just not near the cycle end :)
13:12:26 <SvenKieske> didn't we already agree to discuss this during PTG?
13:13:08 <mnasiadka> Yeah, I think so, just wanted to confirm :)
13:13:26 <mnasiadka> but for example here: https://review.opendev.org/c/openstack/ansible-collection-kolla/+/852240/25/roles/podman/defaults/main.yml#20
13:13:50 <mnasiadka> I see commented out variables in role defaults
13:14:24 <mnasiadka> I'll do some testing of that role and rework it so we keep to the current ,,scheme'' of empty but set variables in defaults :)
13:14:45 <mnasiadka> kevko: mentioned we need python3-podman package in Debian (or fail when installing podman-py outside of venv)
13:15:03 <mnasiadka> Probably for now we should fail, and once the package is in Debian - we can do the same what we do for docker
13:15:10 <mnasiadka> kevko: is that fine?
13:15:15 <kevko> yep
13:15:24 <mnasiadka> great
13:15:38 <mnasiadka> Let's Encrypt - I can tackle that one once podman is in
13:15:46 <SvenKieske> just pointing out the obvious: we could also force a venv for podman, no? it's a new feature?
13:16:24 <mnasiadka> SvenKieske: I think that implies we would force a venv for kolla-ansible and we would need to glue those two things together
13:16:34 <mnasiadka> up until now we haven't forced that on users
13:16:38 <SvenKieske> I'm also fine with the current solution, just wanted to mention that in case it was missed.
13:16:45 <mnasiadka> We could discuss that at the PTG
13:16:49 <SvenKieske> ok
13:17:04 <mnasiadka> bbezak: unless you have cycles to have a look in Let's Encrypt in coming days?
13:17:45 <kevko> i hope LE is written nice now :)
13:18:02 <mnasiadka> kevko: just looking for a thorough review :)
13:18:17 <kevko> i know
13:18:33 <bbezak> sorry, not in 2-3 weeks (going on vacation)
13:19:12 <kevko> regarding python-docker and python-podman ...this module is needed on dest hosts ... so does it mean that you want to install venv on dest hosts and change PYTHONPATH to include venv ?
13:19:27 <mnasiadka> ok then, LE waiting for Podman to merge ;)
13:20:10 <mnasiadka> kevko: I think let's not get into that now, just fail if Debian and not venv ;) We can discuss options at the PTG if the package does not get into Debian
13:20:52 <mnasiadka> jangutter: how are we with etcd bump? is there a patch to add a precheck that Zun does not work at all now?
13:21:03 <mnasiadka> (and a deprecation notice attached)
13:21:29 <mmalchuk> there was a change to drop Zun
13:21:53 <jangutter> I've sent up one last night (sorry got busy with on-call stuff here)
13:21:54 <mmalchuk> https://review.opendev.org/c/openstack/kolla-ansible/+/896593
13:22:36 <mnasiadka> looks nice, thanks
13:22:50 <mnasiadka> kevko, frickler - mind having a look at ^^ ?
13:23:25 <jangutter> Any notes are welcome, since this is mostly going to be a human-interaction thing, if you can find anything that's confusing or should be simpler, don't be afraid to yell!
13:24:14 <opendevreview> Michal Nasiadka proposed openstack/kolla master: Drop docker-py from requirements.txt  https://review.opendev.org/c/openstack/kolla/+/896644
13:24:30 <mnasiadka> ok then
13:24:41 <mnasiadka> what about bumping etcd now? what's the order jangutter ?
13:25:28 <jangutter> Order is the role improvements then the version bump I guess. (doesn't matter much for folks running releases, but it makes it a bit more testable in CI)
13:26:03 <opendevreview> Merged openstack/kolla-ansible master: Add ML2/OVN and ML2/OVS setting checks for neutron  https://review.opendev.org/c/openstack/kolla-ansible/+/895143
13:26:04 <jangutter> but, they're technically totally independent.
13:26:08 <mnasiadka> ok then
13:26:24 <SvenKieske> I yelled :)
13:26:34 <jangutter> Thanks!
13:26:43 <mnasiadka> #link https://review.opendev.org/q/topic:update-etcd-v3.4
13:26:45 <SvenKieske> deprecation != disable would be nice to correct that, it's a pet peeve of mine
13:26:46 <mnasiadka> so that one
13:27:34 <opendevreview> Michal Nasiadka proposed openstack/kolla master: Fix build errors missing podman/docker module  https://review.opendev.org/c/openstack/kolla/+/896586
13:27:53 <mnasiadka> ok, I think that's it for current cycle, I mean enough things to review for a week ;)
13:28:07 <mnasiadka> #topic Additional agenda (from whiteboard)
13:28:36 <mnasiadka> (jsuazo) cinder & galnce S3 support. Ready for merge (?)
13:28:47 <mmalchuk> almost
13:29:00 <mmalchuk> cleanup the commit message and merge
13:30:07 <jsuazo> Will make the change in about a min, would appreciate the +2
13:30:12 <mnasiadka> yes, jsuazo can you apply mmalchuk's comment?
13:30:35 <mnasiadka> added that to RP+1, hopefully it will get more reviews
13:31:05 <opendevreview> Juan Pablo Suazo proposed openstack/kolla-ansible master: Deploy Glance and Cinder Backup with S3 backend support  https://review.opendev.org/c/openstack/kolla-ansible/+/844614
13:31:10 <mnasiadka> (jsuazo) TAAS configurations in kolla
13:31:31 <mnasiadka> ok, that one - I thought we agreed to install TAAS package directly via pip without using additions/sources/whatever?
13:31:46 <mnasiadka> because if we're going to do it like that - we'll need to track versions in sources.py
13:32:15 <opendevreview> Alex Welsh proposed openstack/kayobe master: Add option to skip kolla docker registry login  https://review.opendev.org/c/openstack/kayobe/+/896655
13:32:21 <jsuazo> i think the idea was proposed but not settled on
13:33:07 <jsuazo> we could track versions in sources.py as the TaaS repository has Version branches for each openstack version
13:33:23 <mnasiadka> yes, but it also has a version pin in upper-constraints.txt
13:33:32 <mnasiadka> so it should work properly
13:33:47 <mmalchuk> we are talking about master?
13:34:05 <mmalchuk> imho there is no issue in stable branches?
13:34:08 <opendevreview> Rafal Lewandowski proposed openstack/kolla-ansible stable/2023.1: Add ML2/OVN and ML2/OVS setting checks for neutron  https://review.opendev.org/c/openstack/kolla-ansible/+/896532
13:35:06 <mnasiadka> well, if we could do it in a simple way - we could backport if anybody is interested - it's just adding a package from pypi
13:36:09 <jsuazo> ok, I will look into changing the installation to be done directly with pip and return to you next week
13:36:29 <mnasiadka> great
13:36:44 <mnasiadka> once that's solved - we can have a look on kolla-ansible side
13:37:49 <jsuazo> ok, thnks!
13:38:39 <mnasiadka> (reviewed the k-a patch now as well)
13:38:49 <mnasiadka> ok then
13:38:52 <mnasiadka> #topic Open discussion
13:38:54 <mnasiadka> Anything?
13:38:56 <kevko> yes
13:39:02 <jsuazo> (glance-cinder proposal with latest changes is live)
13:39:04 <kevko> we have rabbitmq images broken :D
13:39:13 <kevko> erlang/rabbitmq incompatibility
13:39:26 <mnasiadka> yes, we'll solve that when backporting the mirror switch
13:39:27 <mnasiadka> no worries
13:39:48 <kevko> you need to ping packages
13:39:51 <kevko> pin
13:39:55 <mmalchuk> http2 merged with an issue
13:39:59 <mmalchuk> quick fix: https://review.opendev.org/c/openstack/kolla-ansible/+/896609
13:40:40 <kevko> and i have a question , do we have some opinion what to do with rabbitmq images if they are not supported anymore ?
13:40:57 <kevko> (i mean for example for stable branches  - if we should fix in stable branch or maintain downstream)
13:41:11 <kevko> mmalchuk: will check
13:42:07 <SvenKieske> you mean no upstream support from rabbitmq or what kind of support?
13:42:12 <mnasiadka> kevko: I had an idea to label rabbitmq container images with a version label and work out a kolla-ansible logic to compare labels and run upgrades if needed (but only where we support rolling ones)
13:42:20 <mnasiadka> I think kevko means EOL versions
13:42:30 <kevko> mnasiadka: yes..
13:42:51 <SvenKieske> so no upstream support, that's what I meant as well :)
13:42:55 <mnasiadka> rolling upgrades are 3.8+
13:43:01 <mnasiadka> so my plan should work
13:43:15 <kevko> i noticed that when i was debugging issue on customer side ... i found the incompatibility with erlang ... (we have 25.3 instead 25.2) ...but moreover ...in log i saw "this is not supported anymore"
13:43:19 <SvenKieske> that sounds like a nice idea
13:43:39 <mnasiadka> but then rolling upgrade works only from 3.8 to 3.9, from 3.9 to 3.10 and so on
13:43:53 <SvenKieske> if we put it behind a feature flag|variable "enable_rolling_rabbitmq_upgrades" or something
13:43:56 <mnasiadka> which might mean we would need to do some tags dance
13:44:03 <mnasiadka> (container image tags)
13:44:04 <SvenKieske> some people might want to pin it for some strange reason
13:44:34 <mnasiadka> I'll do some analysis and come up with a proposal on the PTG
13:45:19 <frickler> just EOL old branches faster *scnr*
13:45:35 <mnasiadka> frickler: we're doing that already
13:45:37 <mnasiadka> :)
13:45:55 <mnasiadka> yoga is going to be EM soon, so might be the situation is solved faster ;)
13:45:59 <frickler> well not fast enough, or we wouldn't have this issue with yoga
13:46:58 <mnasiadka> zed is 3.10 and is EOL as well
13:46:59 <SvenKieske> if upgrades wouldn't introduce breaking changes from time to time it wouldn't really matter how fast stuff get's EOL'ed
13:47:23 <mnasiadka> and antelope is 3.11 and will be EOL in 3 months
13:48:51 <frickler> yeah, anyway, I was only be half serious at most
13:48:52 <kevko> EOL - do you think our users are soo fast in upgrading ? :D
13:48:58 <frickler> *being
13:49:11 <SvenKieske> one day devs will EOL faster than they release new versions :D
13:49:56 <kevko> btw - regarding EOL branches ..
13:50:07 <mnasiadka> Ok then, it seems it's a valid topic, at least RabbitMQ is a bit critical in our setups :)
13:50:20 <kevko> what about new policy to merge patchsets for EOL branchces only with one +2 ?
13:50:26 <kevko> is it possible ?
13:50:28 <mnasiadka> EOL branches do not exist
13:50:47 <mnasiadka> EOL = no branch, only wallaby-eol tag, no merging of anything possible
13:50:49 <kevko> ok - stable branches ..not last
13:50:52 <mnasiadka> you mentioned EM branches
13:50:58 <kevko> but release before for example
13:51:00 <kevko> yes
13:51:31 <kevko> i think it's good idea .. wdyt ?
13:52:39 <mnasiadka> We can discuss on the PTG, I have a contrary idea to not maintain EM at all and just do EOL
13:52:47 <frickler> depends on how many reviewers still care about that branch I guess. and would likely better be documented somehow
13:52:56 <SvenKieske> mhm I think cores should decide, because they have to put in the work. I can't judge if it is avoiding sufficient amount of work to be worthwhile.
13:54:07 <mnasiadka> True, but we should be more specific in how do we maintain "unsupported" branches - I think there was a topic going on to rename EM to Unsupported
13:54:09 <frickler> also EM is being replaced by "Unmaintained", but I didn't look at the details yet and how they apply to deployment projects
13:54:17 <mnasiadka> Ah, Unmaintained
13:54:29 <mnasiadka> https://docs.openstack.org/project-team-guide/stable-branches.html#extended-maintenance
13:54:53 <mnasiadka> I'm wondering if there's a knob called "Kolla does not want to do EM/Unmaintained - move EM branch candidates to EOL directly"
13:55:07 <mnasiadka> But if anybody wants to maintain them - sure, we can discuss that on the PTG
13:55:08 <frickler> https://governance.openstack.org/tc/resolutions/20230724-unmaintained-branches.html is the current resolution
13:55:19 <mnasiadka> From my perspective we have problems merging backports in non-EM branches :D
13:55:42 <mnasiadka> Ah right, frickler congrats on getting elected to TC ;-)
13:56:03 <frickler> thx, that was quite some surprise to me
13:56:29 <mnasiadka> well, you put your name in the contest ;)
13:56:54 <mnasiadka> anyway, I'll add the PTG topics we discussed today to the PTG etherpad
13:57:06 <kevko> cg
13:57:13 <SvenKieske> regarding: "move EM branch candidates to EOL directly" afaik we can do that, this was in fact even possible under the old policy, just not done, iirc.
13:57:24 <mnasiadka> and I'd like to discuss the PTG time slots on next meeting - because not a lot of people are voting on the time slots
13:57:34 <mnasiadka> so we just decide on the next meeting about those slots
13:57:51 <SvenKieske> yeah I'm guilty of not voting for some of the time slots, will do..
13:57:53 <mnasiadka> come with ideas :)
13:58:04 <kevko> link ? :D
13:58:21 <mnasiadka> kevko: do you sometimes read emails?
13:58:37 <kevko> hmm...sometimes :D
13:58:38 <mnasiadka> #link https://lists.openstack.org/pipermail/openstack-discuss/2023-September/035120.html
13:59:16 <SvenKieske> a whole day for kayobe feels excessive ;)
13:59:25 <SvenKieske> just kidding
13:59:52 <kevko> hmm ..i need to try kayobee one day :D
13:59:55 <mnasiadka> https://github.com/rabbitmq/rabbitmq-server/discussions/9553 - I love that RMQ openness - a lot of statements and discussion locked
14:00:15 <mnasiadka> ok, see you next week - thanks for coming :)
14:00:17 <mnasiadka> #endmeeting