13:00:01 <mnasiadka> #startmeeting kolla
13:00:01 <opendevmeet> Meeting started Wed Apr 19 13:00:01 2023 UTC and is due to finish in 60 minutes.  The chair is mnasiadka. Information about MeetBot at http://wiki.debian.org/MeetBot.
13:00:01 <opendevmeet> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
13:00:01 <opendevmeet> The meeting name has been set to 'kolla'
13:00:03 <mnasiadka> #topic rollcall
13:00:04 <mnasiadka> o/
13:00:16 <mmalchuk> \o
13:00:22 <SvenKieske> o/
13:00:23 <mattcrees> o/
13:00:32 <mhiner> o/
13:00:48 <frickler> \o
13:01:14 <kevko> \o/
13:03:52 <mnasiadka> #topic agenda
13:03:52 <mnasiadka> * CI status
13:03:52 <mnasiadka> * Release tasks
13:03:52 <mnasiadka> * Regular stable releases (first meeting in a month)
13:03:52 <mnasiadka> * Current cycle planning
13:03:54 <mnasiadka> * Additional agenda (from whiteboard)
13:03:54 <mnasiadka> * Open discussion
13:03:56 <mnasiadka> #topic CI status
13:04:16 <mnasiadka> any other CI issues that the octavia one that needs slight fixing?
13:04:57 <mnasiadka> hmm, can't see any
13:05:14 <mnasiadka> #topic Release tasks
13:05:41 <frickler> final xena release is done
13:05:52 <frickler> will go into em later this week
13:06:09 <mnasiadka> good, thanks frickler
13:06:33 <mmalchuk> sad and cool news at the same time)
13:06:57 <mnasiadka> once A is out, we might think about marking Wallaby as EOL - given the size of our team
13:07:23 <frickler> +1
13:07:45 <mnasiadka> I think it's time to focus on trying to get podman merged (and other patches with RP+1) - I'll try to make some time this and next week for some focused testing of podman
13:08:09 <mnasiadka> And will go through the release checklist, to make sure most of the preparation tasks are handled
13:08:27 <mnasiadka> #topic Current cycle planning
13:09:19 <mnasiadka> I haven't updated the whiteboard with priorities for Bobcat yet, but the Whiteboard also needs some tidying up after asian translation attack
13:09:31 <kevko> i've spinned master stack so i will probably review this patches .. podman and letsencrypt
13:09:41 <frickler> do we also want to have some kind of freeze, so that we can say other new patches will have to wait until after branching?
13:09:52 <frickler> unless they are bug fixes of course
13:10:27 <mnasiadka> Yes, basically nothing out of RP+1 should be reviewed and merged
13:10:42 <mnasiadka> I mean that doesn't have review priority +1
13:10:57 <mmalchuk> lets merge backports with fixes and freeze
13:11:00 <mnasiadka> and we should focus rather on master branch for a couple of weeks than the backports
13:11:19 <SvenKieske> I see some divergence in opinion here :D
13:11:21 <mmalchuk> Kayobe still lack of reviewers
13:11:57 <mnasiadka> Kayobe always trails behind, that's how it is
13:12:04 <mmalchuk> ok
13:12:18 <mmalchuk> lets merge k/k-a backports with fixes and freeze
13:12:51 <mnasiadka> freeze only affects master branch patches - we don't review those that don't have RP+1
13:12:53 <frickler> the freeze would be for master, stable branches don't need a freeze
13:13:13 <mmalchuk> backports made from the master)
13:13:50 <mmalchuk> some fixes in master still on review
13:13:53 <mnasiadka> if there is anything important that you want review priority on - please post it on the IRC channel, we'll assess and mark with review priority if needed
13:14:03 <frickler> if there is nothing urgent in backports, I'd agree with mnasiadka to focus on master for now
13:14:15 <mnasiadka> so agreed
13:14:17 <mnasiadka> let's move on
13:14:31 <frickler> of course everyone has their own opinion on what is urgent ;)
13:15:12 <mmalchuk> ok, agree
13:15:19 <mnasiadka> #topic Additional agenda (from whiteboard)
13:15:36 <frickler> ah, we still have to look at the other stable releases
13:15:45 <frickler> didn't do them for april yet
13:16:01 <mnasiadka> Ok, let's have a look on the backports
13:16:07 <frickler> or shall we agree to skip for this month?
13:16:31 <mnasiadka> let's skip for this month and revisit in May
13:16:37 <mnasiadka> there's a lot of movement from what I see
13:16:45 <mnasiadka> at least in k-a
13:17:09 <mnasiadka> https://etherpad.opendev.org/p/2023.2-leaderless - retire Sahara, Vitrage, Monasca (already dropped) and Windows support (if we have any remnants of it)?
13:17:14 <mnasiadka> that's the first additional topic
13:17:25 <mnasiadka> so, we have some projects that are getting retired
13:17:33 <mnasiadka> Vitrage is one of them
13:18:01 <mnasiadka> Sahara seems to have survived, but I don't know if it's usable at all in Kolla
13:18:15 <frickler> so we can deprecate now and drop in bobcat?
13:18:22 <mnasiadka> I think so
13:18:24 <kevko> +1
13:18:26 <mmalchuk> lets get rid of them all
13:18:32 <mnasiadka> third one is Winstackers - we've had some Windows support in the past
13:18:43 <mnasiadka> But I don't know if it's still there
13:19:04 <mmalchuk> Windows support useless
13:19:08 <mnasiadka> Anyone volunteering for the effort of writing deprecation renos?
13:19:40 <SvenKieske> I only find two mentions of "windows" in conf.py under the deploy guides section
13:20:27 <SvenKieske> and those are not about windows support, but about windows icon files (.ico)
13:20:37 <mmalchuk> mnasiadka should we write only renos or with code removeing?
13:20:38 <mnasiadka> seems no Windows support, I think we removed it some cycles ago
13:20:39 <frickler> SvenKieske: is that your statement of volunteering? ;)
13:21:09 <mnasiadka> so we need deprecation renos for now, and drop the code in master after A is branched
13:21:30 <SvenKieske> I mean, I guess I could write relnotes, my setup for them is still not complete but that should not take much time (fingers crossed)
13:21:54 <mmalchuk> I can try
13:22:25 <mnasiadka> I think Sven was faster :)
13:22:30 <mmalchuk> ok
13:22:51 <mnasiadka> So, do we want to deprecate Sahara as well?
13:23:04 <mmalchuk> I'll focus on the backports
13:23:10 <SvenKieske> did never use it, so I got no opinion
13:23:17 <mnasiadka> ok, Vitrage for now
13:23:29 <mnasiadka> #action SvenKieske raise patch to deprecate Vitrage (to be dropped in Bobcat cycle)
13:23:32 <mmalchuk> Sahara yes
13:23:50 <frickler> I'd say deprecate sahara, too, wu can still undeprecate if its situation improves
13:24:11 <mnasiadka> #action SvenKieske deprecate Sahara too
13:24:33 <SvenKieske> hum, so questions: do we "deprecate" or do we remove support? that's two different things, no? also: do I only write relnotes, or do I remove code? :D
13:24:50 <mnasiadka> only write renos for now, that it is deprecated and will be dropped in Bobcat cycle
13:24:56 <SvenKieske> okay
13:24:57 <mnasiadka> and we need to drop code in Bobcat cycle
13:25:11 <SvenKieske> because really removing will be a little more work, I see over 300 hits for "sahara" alone
13:25:15 <mnasiadka> (which is sort of now, but not for Kolla, since we're trailing)
13:25:35 <frickler> yes, but likely won't help to prepare the removal now, will be too many merge conflicts
13:25:42 <mnasiadka> Removing code is fun work, don't worry
13:25:51 <mnasiadka> :)
13:25:55 <SvenKieske> yeah. I know :)
13:25:55 <mnasiadka> ok, next topic
13:25:59 <mnasiadka> drop OpenEuler
13:26:06 <mmalchuk> agree
13:26:08 <mnasiadka> so - OpenEuler CI has been failing for a long time
13:26:38 <kevko> +1
13:26:40 <mnasiadka> it was added at the same time as Rocky 8, but since we stopped publishing centos images - that stopped working
13:26:42 <SvenKieske> no replies on the ML on OpenEuler so far, but you maybe could've mentioned that we're deprecating it in the subject line
13:26:43 <mnasiadka> and nobody has picked it up
13:27:06 <mmalchuk> SvenKieske I've replied)
13:27:27 <mnasiadka> well, I could have, but still if somebody would be interested, they would pick it up earlier than my mail
13:27:47 <SvenKieske> mmalchuk: apologies, I meant no replies which wanted to take over maintenance
13:28:03 <mnasiadka> Anyway, surely we should drop the CI job and deprecate
13:28:16 <mnasiadka> What about dropping? Should we wait for Bobcat cycle as well?
13:28:35 <mmalchuk> think no
13:28:36 <SvenKieske> I mean it's broken, so it's practically dead already?
13:28:49 <mnasiadka> practically yes
13:28:55 <frickler> https://review.opendev.org/c/openstack/kolla-ansible/+/879129 is the patch to remove the CI job
13:28:56 <SvenKieske> so +1 for removing
13:28:58 <mnasiadka> it doesn't even go through bootstrap-servers
13:29:12 <mmalchuk> +1 for remove
13:29:18 <SvenKieske> if someone interested shows up it's quickly reverted
13:29:24 <mnasiadka> ok then
13:29:37 <mnasiadka> #agreed to remove OpenEuler this cycle (in addition to deprecating)
13:30:00 <mnasiadka> Anyone wants to remove the code and write the deprecation+drop reno?
13:30:03 <mmalchuk> SvenKieske may be remove line not add comment?
13:31:13 <mmalchuk> frickler sorry
13:31:13 <frickler> mmalchuk: yes, if we want to drop completely, that makes sense
13:31:16 <frickler> I'll update and also add a reno and see what else could be removed
13:31:28 <SvenKieske> I just commented this on the patch, let's move patch discussion to gerrit, no? :)
13:31:29 <mnasiadka> ok then, I'll w-1
13:31:31 <opendevreview> Verification of a change to openstack/kolla-ansible master failed: Stop running broken openeuler job  https://review.opendev.org/c/openstack/kolla-ansible/+/879129
13:31:44 <mnasiadka> ok then, next one
13:31:54 <mnasiadka> (kyarovoy): help needed with https://review.opendev.org/c/openstack/kolla/+/825786, if someone has time - please look at the problems that appear with kolla-build-ansible
13:32:03 <mnasiadka> Anybody did have time to have a look since last week?
13:32:50 <Fl1nt> Hi everyone!
13:32:51 <SvenKieske> well I did look; I really don't get why we support such an old docker release, does it even work in practice?
13:33:15 <mmalchuk> mnasiadka tests?
13:33:19 <mnasiadka> We don't test, so we don't really should support.
13:33:48 <mnasiadka> (about docker version)
13:33:52 <SvenKieske> I agree about missing relno though, but there where unadressed comments anyway, so I didn't want to pile up
13:34:07 <mnasiadka> anyway, this patch broke pushes, so I don't think that without setting up a local env we can fix it
13:34:29 <mnasiadka> If anybody can have a look and help - then please do, I'll leave it for next week
13:34:40 <kevko> mnasiadka: i ran recheck because logs missing :/
13:35:22 <SvenKieske> yeah, maybe the call for help could also be clarified: is it about the failing CI? I'm a big fan of addressing all comments, then look if the CI still fails.
13:35:27 <opendevreview> Verification of a change to openstack/kayobe master failed: Bump up Ansible supported versions to 6.x/7.x  https://review.opendev.org/c/openstack/kayobe/+/878207
13:36:25 <mnasiadka> ok then, no other items in the additional agenda
13:36:41 <mnasiadka> #topic Open discussion
13:36:57 <ihalomi> any news on mounting /run directory?
13:37:14 <kevko> podman ^ ?
13:37:18 <Fl1nt> Can someone review the designate patch?
13:37:26 <ihalomi> yes on podman
13:37:26 <Fl1nt> https://review.opendev.org/c/openstack/kolla-ansible/+/878270
13:37:26 <kevko> Fl1nt which one ?
13:37:33 <Fl1nt> kevko, too quick ^^
13:37:43 <mnasiadka> is it a priority for Antelope?
13:37:47 <kevko> ihalomi: i will review podman today proabbly and i will check
13:37:53 <SvenKieske> regarding mounting /run: afaik that was fixed by mounting all needed subdirs? I actually asked upstream at podman but afaik that's not gonna get fixed
13:37:55 <mnasiadka> it's +131, so it's far from being trivial
13:38:08 <Fl1nt> mnasiadka, as designate is literally broken on any branch yes
13:38:36 <mnasiadka> Fl1nt: As long as you can find two core reviewers that sign up for reviewing that, then sure - feature freeze granted.
13:39:00 <Fl1nt> well hence why I'm asking in here during the weekly meeting ^^
13:39:12 <mnasiadka> And I'm waiting for those two core reviewers :)
13:39:15 <kevko> Fl1nt: i will review also this .. adding to my list ..
13:39:26 <mnasiadka> Do you want to do backports of that to stable branches?
13:39:39 <frickler> designate works fine for me, so it depends on the deployment scenario I guess
13:39:39 <Fl1nt> up to stable/wallaby if possible
13:39:43 <mmalchuk> can someone find two+ core reviewers for Kayobe too?
13:39:52 <Fl1nt> frickler, it works by a hasardous combination.
13:39:55 <kevko> what about https://review.opendev.org/c/openstack/kolla-ansible/+/877413 (multiple ceph files) , this i think can have priority https://review.opendev.org/c/openstack/kolla-ansible/+/874729 masakari
13:39:55 <mnasiadka> Then it needs to be 100% backwards compatible
13:40:04 <mnasiadka> and by just looking at that patch I see it's not.
13:40:09 <Fl1nt> kevko, I've already review this one
13:40:21 <Fl1nt> same feeling as mnasiadka
13:40:26 <mmalchuk> kevko my comments for ceph review not checked
13:40:46 <mnasiadka> kevko: update the reno for masakari, then we can merge it, do I need to write it 3 more times? :)
13:40:52 <ihalomi> okay, so I will create new patchset with mounting of all subdirs needed, should it be defined in one place and then used as variable in all services or should i add it to each service dependent only on what it needs to mount?
13:41:02 <Fl1nt> TBH: this patch about CEPH is weird as it force cross-az cinder usage where you don't want that on a CEPH cluster
13:41:03 <kevko> i will post new patch today in hours ..but there will be only tiny changes and i will resolve comments ...but overall patch is good, isn't it ?
13:41:38 <kevko> mnasiadka, Fl1nt understand , i will ping you in hour maybe
13:41:48 <kevko> masakari thing ? ^^
13:41:53 <Fl1nt> kevko, perfect! I'll be there :D thx!
13:41:54 <mmalchuk> Fl1nt why? one ceph can have multiple pools
13:41:57 <mhiner> ihalomi: i think it should be added to existing podman patch
13:42:49 <kevko> Fl1nt: nope, you just set cross-az: false in config
13:42:51 <Fl1nt> mmalchuk, yes, but AZ isn't the proper segment, it should segment on backends then. AZ is a logical segment such as AZ-1/2/3 with each zone being an aggregate so can represent zone or hw such as HPE/DELL/SUPERMICRO
13:42:52 <mnasiadka> ihalomi: I don't think we mount whole /run on a lot of containers
13:43:19 <Fl1nt> kevko, yes, and it should be the default when using CEPH
13:43:42 <mmalchuk> Fl1nt It's a formality
13:44:02 <kevko> it's not weird , juju has the same implementation
13:44:07 <Fl1nt> mmalchuk, on our setup being HPC/Converged it have impact.
13:44:22 <mmalchuk> there more setups in the world)
13:44:30 <ihalomi> mnasiadka: so listing it in defaults of each service using it is fine?
13:44:42 <SvenKieske> those ceph patches are in general good, I don't get though why they break backward compatibility regarding keyring names and introduce inconsistencies as well.
13:45:29 <Fl1nt> mmalchuk, only three exist, either CEPH as a full storage cluster or converged or mix of both, in all case you do not define AZ on CEPH but on Compute aggregate, each zone can access all backend pools or a set of pools
13:45:29 <mnasiadka> ihalomi: you can have a variable in group_vars/all and just reference the variable where you need it
13:46:08 <mmalchuk> it can break if custom files used before. we shouldn't hardcode order of the backends via list[0] list[1] etc.
13:46:24 <Fl1nt> mmalchuk, exactly.
13:46:40 <mmalchuk> my comments in code about them
13:46:43 <ihalomi> mnasiadka: okay, i will try to do it this week
13:46:44 <kevko> SvenKieske: beacuse it is from beginning bad ... ceph.client.cinder.keyring is   ceph - name of cluster, client - cinder ... if you have azs ..you have to distinguish ceph cluster names
13:47:28 <kevko> SvenKieske : The $cluster metavariable is your Ceph cluster name as defined by the name of the Ceph configuration file (i.e., ceph.conf means the cluster name is ceph; thus, ceph.keyring). The $name metavariable is the user type and user ID (e.g., client.admin; thus, ceph.client.admin.keyring).
13:47:45 <SvenKieske> kevko: but it breaks backward compatibility for just a name cleanup, hard disagree with that
13:48:16 <SvenKieske> kevko: when that should really be fixed, fix it everywhere, not only for 1 variable.
13:48:26 <kevko> SvenKieske: no, it is backward compatible ... it will brae deployment only if you override default values in global.conf
13:48:28 <Fl1nt> SvenKieske, kolla/kolla-ansible try to address a "generic" approach, meaning you need to stick with fact that the service you implement need to stick with based service best practices + variabilization for flexibility and maximum scenario support.
13:48:55 <SvenKieske> the gnocchi key is still named wrong than, anyway let's move patch discussion to gerrit, where it belongs :)
13:49:07 <mnasiadka> yes, patch discussion in gerrit
13:49:07 <mnasiadka> :)
13:49:10 <kevko> SvenKieske and if you overriden defaults by yourself ..you should read reno for upgrades where it is explained ...
13:49:19 <kevko> agree
13:49:23 <mmalchuk> SvenKieske we should have _default variable and the _actual one for backward support
13:49:24 <SvenKieske> kevko: agreed :)
13:51:02 <kevko> last thing - after ceph merge .. kolla-ansible will support azs out of the box ... only aggreagations will be missing ..but it would be small patch to implement ...
13:51:19 <kevko> and that is cool thing
13:51:29 <Fl1nt> kevko, aggregation?
13:51:37 <SvenKieske> question about meeting process: wasn't it the case in the past that the meeting logs are posted to openstack-discuss? did I just miss that mail?
13:51:41 <mmalchuk> small patch?
13:51:58 <ihalomi> about this patchset adding podman to ansible-collection https://review.opendev.org/c/openstack/ansible-collection-kolla/+/852240  do we need more CI test or the ones that are in podman patchset are enough?
13:52:07 <Fl1nt> AZ is placement aggregate, what did I missed there kevko ?
13:52:08 <mnasiadka> there is no such thing as a small patch, unless it's a version bump of a minor component in a Kolla image ;-)
13:52:10 <kevko> yep https://docs.openstack.org/nova/latest/admin/availability-zones.html
13:52:37 <mnasiadka> ihalomi: I think the ones are enough, but as a final solution we need those podman jobs to be run also in ansible-collection-kolla
13:52:46 <mmalchuk> mnasiadka right)
13:53:02 <kevko> Fl1nt: it's more things ..i will propose several patches ..and you will see :)
13:53:07 <mnasiadka> ok then, patch discussions after the meeting or in Gerrit
13:53:14 <mnasiadka> Last stupid question - anybody going to Vancouver?
13:53:24 <Fl1nt> kevko, hum... don't get it, you create an AZ by associating placement metadata with host aggregates.
13:53:25 <kevko> nope :(
13:53:33 <SvenKieske> it's sad that there are no virtual tickets for vancouver
13:53:39 <ihalomi> mnasiadka: okay so no need to work on that for now
13:53:44 <kevko> Fl1nt: i will write you DM if you are curious
13:53:52 <Fl1nt> mnasiadka, nope, waiting for EU based one ^^
13:53:56 <frickler> it is sad that the foundation still does in-person events at all
13:54:00 <Fl1nt> kevko, mosdef!
13:54:34 <Fl1nt> frickler, in-person is cool, but they should do smaller one more often and spread everywhere as GDG
13:54:42 <mnasiadka> Ok, Kolla User Forum session was accepted, but probably will be lower attendance than last time in Berlin
13:54:55 <mnasiadka> Anyway - I need to run
13:55:04 <mnasiadka> thanks for attending and lively conversations
13:55:05 <Fl1nt> mnasiadka, see ya
13:55:06 <mnasiadka> See you next week!
13:55:09 <mnasiadka> #endmeeting