13:00:14 <mnasiadka> #startmeeting kolla
13:00:14 <opendevmeet> Meeting started Wed Jun 19 13:00:14 2024 UTC and is due to finish in 60 minutes.  The chair is mnasiadka. Information about MeetBot at http://wiki.debian.org/MeetBot.
13:00:14 <opendevmeet> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
13:00:14 <opendevmeet> The meeting name has been set to 'kolla'
13:00:17 <mnasiadka> #topic rollcall
13:00:18 <mnasiadka> o/
13:00:24 <SvenKieske> o/
13:00:25 <mmalchuk> o/
13:00:33 <r-krcek> o/
13:00:35 <mhiner> o/
13:03:08 <opendevreview> Michal Nasiadka proposed openstack/kayobe master: seed-vm/infra-vms: Add support for EFI and q35  https://review.opendev.org/c/openstack/kayobe/+/921660
13:03:23 <mnasiadka> #topic agenda
13:03:23 <mnasiadka> * CI status
13:03:24 <mnasiadka> * Release tasks
13:03:24 <mnasiadka> * Regular stable releases (first meeting in a month)
13:03:24 <mnasiadka> * Current cycle planning
13:03:25 <mnasiadka> * Additional agenda (from whiteboard)
13:03:25 <mnasiadka> * Open discussion
13:03:27 <mnasiadka> #topic CI status
13:03:36 <mnasiadka> Kolla/Kolla-Ansible seems green
13:03:43 <mnasiadka> Kayobe SLURP jobs are still failing
13:03:49 <mnasiadka> anything I missed?
13:04:34 <SvenKieske> kolla build failure rate seems up, at least according to https://grafana.opendev.org/d/c0d59dad13/kolla-failure-rate?orgId=1 ?
13:04:56 <SvenKieske> kolla-ansible is broken in grafana (I guess for some time again, I should check more often)
13:06:02 <mnasiadka> let me check kolla periodics
13:06:43 <mnasiadka> last failures from 15th of June in Kolla
13:07:04 <SvenKieske> mhm, maybe just grafana acting up
13:07:08 <mnasiadka> anyway, let's continue
13:07:13 <mnasiadka> #topic Release tasks
13:08:46 <mnasiadka> R-17 has passed, according to Release Management schedule - we should switch to master branches from 2024.1
13:09:03 <mnasiadka> Something like https://review.opendev.org/c/openstack/kayobe/+/791764
13:09:13 <mnasiadka> Any volunteers to do this?
13:10:01 <mmalchuk> I can
13:10:29 <SvenKieske> ty :)
13:10:30 <mnasiadka> ok the
13:10:32 <mnasiadka> n
13:10:34 <mnasiadka> let's move on
13:10:42 <kevko> \o
13:10:42 <mnasiadka> #topic Current cycle planning
13:11:16 <mnasiadka> I started some work on the Ubuntu Noble LTS support and Ansible bump
13:11:34 <mnasiadka> Ubuntu nodes do OOM on new Ansible, and some other fantastic things - so having fun
13:12:10 <SvenKieske> ah we just talked about this internally today (noble). notice afaik noble doesn't support upgrades from 22.04 until 24.04.1 release
13:12:11 <mnasiadka> Does anybody have any other high priority features that need discussing?
13:12:44 <mnasiadka> I'm not interested in in-place upgrades now, we just need to get that working and backport this to C
13:13:04 <frickler> the oom issue should be fixed with 2.17.1, see the comments in your patch
13:13:26 <kevko> as always ..me :) ..just find some time to review again my patches
13:14:13 <mnasiadka> Yeah, and hundreds of other patches :)
13:14:23 <mnasiadka> Let's go to additional agenda
13:14:30 <mnasiadka> #topic Additional agenda (from whiteboard)
13:14:36 <kevko> yeah, but i am actively applying changes regarding comments :)
13:14:49 <mnasiadka> (SvenKieske: please review patch for broken keystone federation backend https://bugs.launchpad.net/kolla-ansible/+bug/2058656
13:14:49 <mnasiadka> trivial patch at https://review.opendev.org/c/openstack/kolla-ansible/+/913908 needs Cores <--still needs input
13:14:49 <mnasiadka> https://review.opendev.org/c/openstack/kolla-ansible/+/921381/comment/56b11f74_f98925e8/ (rmq startuptime high, reason found, please share your opinion)
13:15:36 <SvenKieske> yeah the first two are rather getting old, they really don't need any discussion in the meeting imho, just feedback on the patches itself.
13:16:04 <SvenKieske> the third one I don't know, I really would like to have some more opinions on it, but could also be discussed on the patch itself
13:16:12 <mnasiadka> My question is - do we have ANY CI job that at least configures keystone to do federation?
13:16:26 <SvenKieske> let me check..
13:17:07 <opendevreview> Maksim Malchuk proposed openstack/kolla master: Revert "[release] Use Caracal sources by default"  https://review.opendev.org/c/openstack/kolla/+/922318
13:17:07 <mnasiadka> Because if we don't, we're going to see those issues every now and then
13:17:19 <SvenKieske> true
13:17:34 <SvenKieske> at least from quick grepping I don't find any tests
13:17:55 <SvenKieske> that being said, a bugfix is a bugfix. we have many bugfixes where you don't ask for tests, even if there are none
13:18:19 <kevko> we should add some ...
13:18:33 <SvenKieske> and ftr I agree we should maybe focus more on tests, but I would appreciate it if this pressure would be applied equally, at least it doesn't feel to be applied equally, from my pov.
13:18:56 <SvenKieske> but ymmv :)
13:19:03 <opendevreview> Mark Goddard proposed openstack/kolla-ansible master: Add support for customising Neutron physical network names  https://review.opendev.org/c/openstack/kolla-ansible/+/922320
13:19:23 <mnasiadka> Well, I'm not going to set up an AIO with federation enabled to check if that fixes anything
13:20:50 <mnasiadka> commented on the patch
13:21:38 <mnasiadka> SvenKieske: what about that RMQ thing?
13:22:24 <SvenKieske> the path thing?
13:22:57 <SvenKieske> well the startup time is raised, because it is computed everytime rmq starts, so we should move it either to build time (in the dockerfile in kolla) or directly hardcode it
13:23:43 <SvenKieske> not sure though how often this changes, I just had a look at the otp repo where imho the ssl libs erlang sources reside, there is already version 11 and they do frequent releases, but we do seem to have version 9.4 somewhere..
13:24:01 <SvenKieske> if that changes frequently we might not want to update this string manually in the dockerfile
13:24:15 <mnasiadka> well, that's erlang version?
13:24:33 <mnasiadka> ah, ssl lib in erlang
13:24:40 <SvenKieske> so I think the best compromise would be to let the erl autodiscovery run at build time, so it doesn't run every rmq start, but we don't need to patch for every ssl lib minor version bump?
13:25:35 <SvenKieske> from my pov that makes sense, but I'd really like to have some opinions of seasoned maintainers there, so we don't need to rewrite the patch in one or two weeks :)
13:25:55 <SvenKieske> I discussed this briefly with Matus (patch author) in private.
13:26:18 <mnasiadka> well, adding that to rabbitmq-env.conf - which we template out in kolla-ansible?
13:26:36 <SvenKieske> this really did affect the build pipeline which sometimes would just timeout
13:27:05 <SvenKieske> that's another option, but I'm not sure it fixes anything? doesn't this run on every start as well? or do you mean hardcoding it there?
13:27:17 <mnasiadka> according to RMQ docs - we need to add that to rabbitmq-env.conf
13:27:31 <mnasiadka> and if we template out the whole file in kolla-ansible
13:27:42 <SvenKieske> let me look. wait ok if we template it, it's only run once, seems also fine
13:27:43 <mnasiadka> then adding that in Dockerfile isn't going to get us anywhere
13:27:50 <SvenKieske> true
13:28:38 <SvenKieske> I'm really not an expert, this patchset is the first time I've even seen this option, just looked it up myself in upstream docs.
13:28:38 <mnasiadka> So unless that's a big burden - I would say it should be added in kolla-ansible
13:29:14 <mnasiadka> probably just spawning a sidecar container with rabbitmq to get that information out of it - and then use the output to be templated out to rabbitmq-env.conf - only when we enable rabbitmq tls
13:29:17 <SvenKieske> it's fine either way :) it just doesn't make sense to compute it on every startup and make startup slower and this leads to CI pipeline timeouts in some tests :)
13:29:43 <mnasiadka> Anyway - any volunteer to do this?
13:30:10 <SvenKieske> I think Matus can implement it in the same patchset if he has some guidance, should be possible with kolla_toolbox, no?
13:30:51 <SvenKieske> or should this go in a separate patch? afaik the ssl stuff is only needed for this patch? so can be developed together?
13:30:56 <mnasiadka> Don't we want to get that backported?
13:31:06 <mnasiadka> We have RabbitMQ TLS working today, don't we?
13:31:23 <SvenKieske> this is only inter node tls, so the inter cluster traffic
13:31:25 <mnasiadka> or is that only for internode TLS?
13:31:28 <mnasiadka> ok
13:31:33 <mnasiadka> then incorporate that in the current patch
13:31:35 <SvenKieske> I don't know if it affects other TLS settings actually
13:31:44 <mnasiadka> well, maybe it's good to check ;-)
13:31:51 <SvenKieske> yeah, will do
13:31:59 <mnasiadka> but RMQ docs rather mention inter-node
13:32:12 <mnasiadka> anyway
13:32:13 <mnasiadka> let's move on
13:32:23 <mnasiadka> (r-krcek): please review https://review.opendev.org/c/openstack/kolla-ansible/+/914997
13:33:17 <opendevreview> Maksim Malchuk proposed openstack/kayobe master: Revert "[release] Use OpenStack 2024.1 release"  https://review.opendev.org/c/openstack/kayobe/+/922322
13:34:06 <mnasiadka> oh crap, love that kind of changes
13:35:12 <mnasiadka> Will have a look
13:35:26 <mnasiadka> #topic Open discussion
13:35:29 <mnasiadka> Let's move on
13:35:32 <mnasiadka> Actually I have a topic
13:37:13 <mnasiadka> Since we only have two core-reviewers that are really active outside of StackHPC - I'd like to propose that if a change has one +2 from SHPC - and a second review has not landed for 2 weeks (after setting RP+1) - I'd like to merge such patches with single company approval.
13:37:38 <mnasiadka> frickler, kevko: What do you think about that?
13:38:58 <mmalchuk> please review https://review.opendev.org/c/openstack/kayobe/+/921627 as discussed I've made the followup
13:39:40 <kevko> i am not sure if i understand
13:39:51 <frickler> mnasiadka: I guess that's fine for me, even 1 week at most should be fine
13:39:53 <mmalchuk> and new for Kayobe: https://review.opendev.org/c/openstack/kayobe/+/921628
13:40:07 <mmalchuk> summer lack of cores?
13:40:14 <mmalchuk> again?
13:40:23 <mnasiadka> kevko: currently we have a rule - if the author is from SHPC - we can't merge it with two core reviewers from SHPC
13:40:43 <mnasiadka> We have a lot of changes, that are stalled due to missing core review from outside of SHPC
13:40:52 <mmalchuk> mnasiadka lets add cores
13:40:55 <mnasiadka> Those have usually RP+1 - and it does not help
13:40:58 <mnasiadka> mmalchuk: shh
13:41:02 <frickler> well the role is for any company, not specific to SHPC, just to avoid confusion
13:41:07 <mnasiadka> yes
13:41:38 <mmalchuk> frickler not only SHPC use Kolla projects
13:41:40 <kevko> i have the same problem ... what about trade +2 so you will give me +2 for my review and I will find  a time for your review :D
13:41:43 <mnasiadka> so - in order to not let changes to rot - I'm proposing to merge patches with single company approval 2 weeks after the first +2 - if a second +2 from a separate company does not come in
13:42:06 <mnasiadka> kevko: show me your patches that already have one +2 :)
13:43:03 <SvenKieske> I was not asked, but maybe require at least one +1 from another company? that should be a pretty low bar, so maybe not worth spelling it out, though. ¯\_(ツ)_/¯
13:43:40 <mnasiadka> So, there's another thing that irritates me most - if I can
13:43:52 <mnasiadka> Adding +1 to a patch after it gets merged - that's to you SvenKieske and mmalchuk ;-)
13:43:59 <kevko> mnasiadka: well this is another problem ..how can I get some reviewer if you guys in SHPC just giving +2 to each other ...
13:44:12 <kevko> but no problem ..i am ok with it btw
13:44:42 <kevko> faster merging ..is always better ..it's still master ..it can be reverted ..it can be fixed by another patch if needed
13:45:26 <mnasiadka> kevko: sure, everybody has the same problem - it's even hard to find two SHPC cores having time to review a patch ;-)
13:45:48 <SvenKieske> mnasiadka: well sometimes I review a patch, give +1, then there is last minute change like typo in commit message and immediately +2 +1 workflow and merged and my review is "gone". I think it's at least a decent signal to re-review the minor change then and say "I also did look at it"
13:45:52 <mnasiadka> But I'll try to have a look in your patches next week - last two weeks have been extremely hectic on my side
13:45:58 <SvenKieske> if that's annoying I can leave it be I guess
13:46:19 <mnasiadka> I think it's useful if there's a comment "you missed that", or "please follow up with a fix to this"
13:46:20 <kevko> Or we should create some queue or something ...
13:46:25 <mnasiadka> but just +1 is a bit irritating :)
13:46:33 <kevko> for example here is reaaaallly trivial and important CI fix
13:46:35 <kevko> https://review.opendev.org/c/openstack/kolla-ansible/+/919939
13:46:39 <mnasiadka> kevko: we have a queue, just add RP+1 :)
13:47:11 <opendevreview> Michal Nasiadka proposed openstack/kolla-ansible master: CI: Use 2024.1 as previous_release  https://review.opendev.org/c/openstack/kolla-ansible/+/919939
13:47:53 <SvenKieske> maybe a good reminder that not only mnasiadka can RP+1; not sure if everybody knows, but mere mortals (non cores) can't RP+1 (currently)
13:48:07 <kevko> because ...without this patch for example we are testing totally something different as we should ... because now it's tested from 2023.2 to -> current uploaded patch
13:48:15 <mnasiadka> yup
13:48:19 <mnasiadka> I even updated that patch :)
13:48:25 <kevko> mnasiadka: I know
13:48:27 <mnasiadka> frickler: if you can have a look in ^^
13:48:41 <mnasiadka> kevko: as I said, I've been extremely busy - it should be better next week ;)
13:48:42 <SvenKieske> this would also result in less clicks for backport voting btw, not sure how many reviews it needs, it has gotten quite some attention already: https://review.opendev.org/c/openstack/project-config/+/920472
13:49:03 <mnasiadka> SvenKieske: that is project-config, I have no power over that :)
13:49:10 <kevko> SvenKieske: well, I also added some RP +1 ...but somebody just deleted it :D
13:49:12 <SvenKieske> frickler? ;)
13:49:25 <SvenKieske> kevko: lol ok, didn't notice, too bad :)
13:49:27 <frickler> hmm, "branches: none" is a weird idea, I have to think about that
13:49:56 <mnasiadka> other option is to remove the config at all
13:50:04 <mnasiadka> and revert in E
13:50:13 <mnasiadka> but we would need to add some docs to the release management guide
13:50:39 <mnasiadka> Feel free to comment with a better solution
13:50:50 <mnasiadka> we could have a CI var "master_is_slurp"
13:50:53 <mnasiadka> maybe that's better
13:51:11 <mnasiadka> if Zuul can honor that
13:51:20 <SvenKieske> that sounds decent, if it works.
13:51:44 <frickler> the thing is, the job might still run if it is defined somewhere else, like from stable/2024.1
13:51:46 <SvenKieske> I really would like a handy list which releases are slurp, didn't sink into my brain just yet
13:52:08 <kevko> SvenKieske: https://releases.openstack.org/ :D ?
13:52:10 <frickler> *.1 is slurp, *.2 isn't. slurp == odd
13:52:36 <mnasiadka> frickler: basically I would be happier if we run slurp whole cycle
13:52:39 <SvenKieske> frickler: ah that's a good shorthand!
13:53:01 <SvenKieske> and I didn't know it's mentioned on the release page as well, thx!
13:53:48 <mnasiadka> as in when it's master branch, not just when we do rc1 and find out slurp does not work
13:55:07 <frickler> but for current master, from where would you upgrade? from 2024.1 it is just normal upgrade jobs, don't need to duplicate those
13:55:22 <frickler> from 2023.1 is too old, we don't want to support that
13:55:37 <frickler> from 2023.2 ... weird, might not be supported by services
13:55:44 <mnasiadka> well, when master is 2025.1
13:55:54 <mnasiadka> we should be doing slurp from 2024.1 to master (to be 2025.1)
13:55:55 <mnasiadka> right?
13:56:01 <frickler> yes
13:56:16 <frickler> so re-enable as soon as 2024.2 is branched
13:56:39 <mnasiadka> yeah, was thinking if we can have jobs running when some variable is true
13:56:45 <mnasiadka> so we would only change that variable
13:57:17 <mnasiadka> but still setting up slurp per new release is going to take some work
13:57:24 <frickler> I can check with zuul ppl whether that would be possible
13:57:25 <mnasiadka> so maybe we just remove jobs from project.yaml
13:57:36 <mnasiadka> and add them back when master is slurp
13:57:43 <mnasiadka> and just document that in our Release Management docs
13:57:46 <mnasiadka> I'm fine with any :)
13:58:00 <frickler> yes, that would be the usual approach that openstack-zuul-jobs is taking
13:58:10 <mnasiadka> ok, so let's do that
13:58:21 <frickler> could even add them all in one project-template and just add/remove that one maybe
13:58:40 <opendevreview> Michal Nasiadka proposed openstack/kolla-ansible master: CI: Use 2024.1 as previous_release  https://review.opendev.org/c/openstack/kolla-ansible/+/919939
13:58:52 <mnasiadka> that also makes sense
13:59:07 <mnasiadka> I'll have a go at writing the docs and trying out project template
13:59:24 <mnasiadka> thanks
13:59:31 <mnasiadka> ok, we're out of time :)
13:59:38 <mnasiadka> Thank you all for coming - see you next week :)
13:59:40 <mnasiadka> #endmeeting