14:00:39 <bbezak> #startmeeting kolla 14:00:39 <opendevmeet> Meeting started Wed Mar 27 14:00:39 2024 UTC and is due to finish in 60 minutes. The chair is bbezak. Information about MeetBot at http://wiki.debian.org/MeetBot. 14:00:39 <opendevmeet> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 14:00:39 <opendevmeet> The meeting name has been set to 'kolla' 14:00:46 <bbezak> #topic rollcall 14:00:49 <mmalchuk> o/ 14:01:09 <kevko> \o/ 14:01:10 <SvenKieske> o/ 14:01:13 <mattcrees> o/ 14:01:22 <r-krcek> o/ 14:01:32 <mhiner> o/ 14:01:48 <jovial> o/ 14:01:49 <frickler> \o 14:02:35 <bbezak> #topic agenda 14:02:37 <bbezak> * Roll-call 14:02:37 <bbezak> * Agenda 14:02:37 <bbezak> * Announcements 14:02:37 <bbezak> * Review action items from the last meeting 14:02:37 <bbezak> * CI status 14:02:38 <bbezak> * Release tasks 14:02:38 <bbezak> * Regular stable releases (first meeting in a month) 14:02:40 <bbezak> * Current cycle planning 14:02:40 <bbezak> * Additional agenda (from whiteboard) 14:02:42 <bbezak> * Open discussion 14:03:26 <bbezak> #topic CI status 14:03:53 <bbezak> watermelon? 14:04:09 <frickler> red on the inside? ;) 14:04:11 <SvenKieske> watermelon? :D 14:04:15 <SvenKieske> ah! lol 14:04:18 <bbezak> :) 14:04:20 <SvenKieske> didn't get that 14:04:37 <mmalchuk> green outside, red inside))) lol 14:04:43 <bbezak> exactly 14:04:52 <bbezak> not so bad nowadays I think 14:05:03 <SvenKieske> well at least kolla build failrate seems flat since 24.03. 14:05:30 <SvenKieske> wasn't the dashboard supposed to be fixed? I see "no data" again: https://grafana.opendev.org/d/c0d59dad13/kolla-failure-rate?orgId=1 14:05:55 <SvenKieske> for k-a source graphs, kolla graphs are fine 14:06:06 <bbezak> I think it was working last time I checked 14:06:14 <frickler> I didn't look at it in ages, no idea 14:06:32 <SvenKieske> I'm pretty sure as well it worked, I'm to blame for not looking often enough I guess. :D 14:06:49 <bbezak> ok, let's look into that 14:07:19 <frickler> I haven't seen too many complaints from kevko recently, so CI can't be too red in general, I'd say ;) 14:07:39 <bbezak> yeah, I think it looks good 14:07:58 <kevko> :D 14:08:13 <kevko> centos build is broken 14:08:21 <kevko> but it's non-voting 14:08:23 <SvenKieske> mhm, can I ask something about CI? I know we plan to extend ARM support downstream, ARM pipeline still seems rather flaky 14:08:45 <SvenKieske> and is non voting, if it's in good shape now, could we make it voting? 14:08:58 <frickler> seems there was an issue with centos mirrors earlier today, jrosser was also complaining, should be better now 14:09:28 <frickler> for ARM builds we only have a single cloud provider, that's mostly why we consider it not too reliable 14:10:29 <bbezak> #topic Release tasks 14:10:43 <bbezak> we're in R-1! 14:11:05 <mnasiadka> People running and screaming 14:11:13 <bbezak> :) 14:11:17 <bbezak> https://docs.openstack.org/kolla/latest/contributor/release-management.html#r-1-prepare-kolla-kolla-ansible-for-rc1-stable-branch-creation 14:11:43 <frickler> bit early for branching, isn't it? 14:11:53 <mnasiadka> It says prepare 14:12:04 <mnasiadka> And we will be prepared for next 3 months 14:12:10 <bbezak> let's start to prepare then 14:12:17 <bbezak> I can take a look 14:12:33 <SvenKieske> well the UCA and RDO stuff got merged afaik 14:12:45 <mnasiadka> So let’s have a look what is missing 14:12:53 <mnasiadka> And we need a volunteer to fix Octavia 14:13:10 <kevko> octavia ? what is with octavia ? 14:13:17 <kevko> *wrong 14:13:19 <SvenKieske> do we have a bugreport for octavia? I have only a fuzzy memory what exactly is broken 14:13:36 <mnasiadka> kevko: CI is failing on cascade delete of LB 14:13:45 <kevko> aaaaa, i saw this 14:13:47 <mmalchuk> maybe forgot anything in A-C-K ? again 14:14:41 <bbezak> any volunteer to take a look into octavia in CI ? 14:14:45 <SvenKieske> now, is this a release task? or just regular bugfix? 14:14:57 <mnasiadka> Exploratory surgery 14:15:00 <bbezak> regular bugfix needed for release 14:15:01 <bbezak> :) 14:15:02 <SvenKieske> :D 14:15:08 <kevko> bbezak: i will probably because i am playing with ci and now everything is almost working.... so i am playing around ... 14:15:21 <bbezak> cool 14:15:29 <bbezak> shout if you need help 14:15:39 <kevko> ok .. 14:15:48 <SvenKieske> I got a question regarding the release tasks posted above, it has this gem: "The release notes for each project should be tidied up" and I have no idea what that means even after following the review examples there. 14:16:17 <mnasiadka> It means read through all renos, fix obvious errors, remove duplicates 14:16:18 <SvenKieske> it seems there are mainly old release notes being deleted? I'm not really sure, is this obsolete? 14:16:44 <mnasiadka> So it looks readable for a random person 14:16:53 <SvenKieske> ah okay. weird. I would think this is part of normal code review, but okay, thanks for the explainer :) 14:17:15 <mnasiadka> We have sort of a rule to not obsess over renos in code review 14:17:21 <mnasiadka> And fix it close to release 14:17:36 <jovial> And mistakes do sneek in :) 14:17:38 <mnasiadka> To improve the overall merge flow or something like that 14:18:03 <frickler> some things are also easier to spot with a global view, like consistent formatting, e.g. use ``code`` where applicable 14:18:07 <SvenKieske> ok, I guess I missed those fixes then the last few releases :) thanks for the explanation, I just didn't know how to interpret "tidied up". 14:18:23 <mnasiadka> Ok, I’d like to discuss the ptg schedule if bbezak will permit 14:18:34 <bbezak> I'll think about it 14:18:50 <mnasiadka> I can’t do any other day that Monday, bbezak is on vacation 14:18:53 <bbezak> ok, granted 14:18:54 <bbezak> #topic PTG planning 14:19:11 <mnasiadka> So my proposal is to do Kolla/Kolla-Ansible on Monday for 4 hours 14:19:44 <mnasiadka> And follow up on not discussed topic on the week after if needed 14:20:31 <mnasiadka> Unless frickler or kevko feel an urgent need to drive the Kolla PTG on Tue 14:20:36 <kevko> hmm, we have state holiday :/ 14:20:51 <kevko> but i will probably join .. 14:20:57 <mnasiadka> kevko: on Monday? 14:21:14 <kevko> let me check 14:21:28 <kevko> i am not sure 14:21:29 <SvenKieske> that would be April, 8th 14:21:39 <frickler> I'm fine with skipping Tue 14:22:27 <kevko> okay, we don't :) 14:22:35 <mnasiadka> Ok, so that’s agreed - Monday only 14:22:40 <kevko> +1 14:22:45 <mnasiadka> jovial: want to move Kayobe? 14:22:53 <mnasiadka> It’s currently planned for Thursday 14:23:09 <jovial> I would prefer Wednesday if we did move it. But does that work for others? 14:24:08 <SvenKieske> does somebody adjust the ptg schedule then? so people are informed? 14:24:59 <mnasiadka> I will adjust and send mail to Ml 14:25:18 <mmalchuk> all this because we skip doodle service) 14:25:23 <mnasiadka> jovial: Kayobe is mainly SHPC - just make sure key people will attend and I’ll move 14:25:39 <jovial> I guess if Kayobe was moved to the Tuesday then we'll avoid the normal kolla IRC meeting 14:25:48 <mnasiadka> mmalchuk: doodle made no sense because you all chose the same slots we used for ages 14:26:12 <mmalchuk> maybe or maybe not 14:27:01 <mmalchuk> anyway we change slots 14:27:39 <bbezak> ok, anything else PTG related? 14:28:37 <bbezak> as a reminder a link to etherpad - https://etherpad.opendev.org/p/kolla-dalmatian-ptg 14:30:02 <bbezak> let's move on 14:30:18 <SvenKieske> #Link https://etherpad.opendev.org/p/kolla-dalmatian-ptg 14:30:18 <bbezak> #topic Current cycle planning 14:31:27 <bbezak> does anybody has something to discuss about outstanding tasks? 14:32:09 <bbezak> I was overloaded with customer related tasks recently, so no much update from me I'm affraid 14:32:40 <kevko> I want to ask to merge init-runonce modification to stable branches 14:32:57 <kevko> :) 14:33:28 <SvenKieske> well looking at the caracal tasks not that many are done afaik? 14:34:17 <SvenKieske> anybody has any news on the next ubuntu release? we wanted to build that in for C release 14:34:47 <kevko> 25.4 ? :D 14:34:53 <mmalchuk> nope... silent 14:34:55 <SvenKieske> no, 24.04. 14:35:00 <bbezak> kevko: I've seen a bit discussion the other day. I don't think it is a good idea to modify stable branches without doing the same to master? 14:35:08 <kevko> google saying Ubuntu 24.04 LTS, codenamed Noble Numbat, is scheduled for release on April 25, 2024. 14:35:29 <mmalchuk> SvenKieske ask again afer 25 apr 14:35:39 <mmalchuk> after* 14:35:56 <kevko> bbezak: yes, i want to merge also to master ...same patch ...but it's needed also to merge to stable branches ..as upgrade jobs are checkout ing previous release ... 14:36:02 <SvenKieske> but why was it put on the agenda then? I mean the release dates of ubuntu are pretty well known in advance? weird. 14:36:37 <kevko> bbezak: moreover, it's only runonce script which is only suggested to use as per documentation ...normally it's not used I would say ..with end user 14:37:21 <bbezak> SvenKieske: I think it was just to prepare images building for now. And caracal is a good place for that as it is a SLURP release 14:37:28 <kevko> bbezak: now i hacked ci :D and download script from my github repo to pass upgrade jobs :D 14:37:51 <bbezak> 24.04 feature will be probably backported to caracal anyway at some point 14:38:03 <SvenKieske> ok 14:38:14 <mmalchuk> btw about CI we have an issue with docs site preview. anyone know this? 14:39:06 <SvenKieske> meta: can we maybe clean up the whiteboard? it still has listed e.g. train PTG; afaik a link to some archived document would be enough there, no? the Whiteboard is 880 lines long now and hard to navigate.. 14:39:46 <SvenKieske> sorry, maybe off topic for current cycle planning 14:40:14 <frickler> mmalchuk: where do you see that issue? 14:40:28 <mmalchuk> preview didn't work 14:40:39 <mmalchuk> on any last change 14:40:41 <bbezak> SvenKieske: I'll add it to notes 14:40:49 <mmalchuk> I'll show you... moment 14:41:21 <SvenKieske> bbezak: ty, I can also volunteer to do cleanup or create some archive somewhere (different etherpad) as long as I know what can be cleaned up and what should stay. 14:41:28 <frickler> wfm https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_33b/911417/12/check/openstack-tox-docs/33b87a9/docs/ 14:42:04 <SvenKieske> mmalchuk: maybe some storage issue? did you get an (HTTP) error? anything? 14:42:20 <mmalchuk> oops... fixed. looks like infra knows and fixed 14:43:07 <mmalchuk> it was an issue with listing 403 forbidden with banner 14:43:26 <mmalchuk> anyway fixed... lets move ahead 14:43:33 <bbezak> #topic Additional agenda (from whiteboard) 14:43:44 <bbezak> I can see mostly a review requests there 14:43:58 <bbezak> anything to add apart of that? 14:44:09 <SvenKieske> yes :) I already asked this week but I guess if you don't highlight usernames nobody has time :P :D 14:44:21 <mhiner> yes, plus Ivan still has that issue with Docker version on Debian 14:44:22 <bbezak> mhiner, r-krkcek 14:44:35 <bbezak> sorry :) r-krcek 14:45:17 <mhiner> other than that, it's just reviews from me 14:45:26 <SvenKieske> so I'll explicitly mention it here: mnasiadka, frickler: https://review.opendev.org/c/openstack/kolla-ansible/+/855498 please come to a solution where to test ovn-exporter, currently I made it work in the ovn scenario and I honestly would prefer if it stays there, because these tests are pretty brittle 14:46:08 <SvenKieske> meaning it didn't work at all in prometheus-opensearch scenario for many reasons. 14:46:20 <r-krcek> For me its the reviews and one question, but wanted to ask that in the open debate. 14:47:00 <bbezak> #topic Open discussion 14:47:17 <SvenKieske> r-krcek: seems you need to rebase the "add sysctl role change" https://review.opendev.org/c/openstack/kolla-ansible/+/912351 14:47:19 <r-krcek> I noticed that we have jinja filter for finding out that service is enabled and mapped to host. However, it looks like it is used only with certain services. Is there a specific reason for not using it everywhere or was is just because of lack of time to do it? The filter: https://review.opendev.org/c/openstack/kolla-ansible/+/665922 14:47:58 <mmalchuk> lets discuss https://bugs.launchpad.net/kolla/+bug/2042954 14:48:14 <r-krcek> SvenKieske: oh, sorry. Ill do that right after the meeting. 14:48:19 <mmalchuk> imho https://review.opendev.org/c/openstack/kolla/+/913584 not needed and https://review.opendev.org/c/openstack/kolla/+/904781 need revert 14:49:26 <mmalchuk> the issue appears in kernel 5.18+ but we have 5.4 and 5.15 maximum 14:50:04 <mmalchuk> the issue not reproduced on Focal and Yoga 14:50:26 <SvenKieske> mmalchuk: case in point: I know people running HWE kernels on jammy, which bumps you up til 6.5 kernel :) 14:50:40 <mmalchuk> HWE is a 5.15 14:50:42 <opendevreview> Roman Krček proposed openstack/kolla-ansible master: Update cell0 database connection https://review.opendev.org/c/openstack/kolla-ansible/+/910924 14:51:08 <frickler> 22.04-hwe is 6.5 14:52:13 <mmalchuk> should we support HWE on Jammy in Yoga? Jammy only for transition to Zed 14:52:32 <frickler> yoga is unmaintained, we should not care at all 14:52:37 <mmalchuk> so 14:52:41 <SvenKieske> frickler: maybe we don't have this issue because we have newer openstack? 14:52:51 <SvenKieske> didn't look into it what the actual issue is 14:53:03 <mmalchuk> I've dig deeper 14:53:20 <mmalchuk> the issue only on upstream kernel with old libs 14:53:20 <SvenKieske> funnily enough the bugreport doesn't mention what the bug is, just that there is one. 14:54:00 <SvenKieske> ah this is the original bug: https://bugs.launchpad.net/ubuntu/+source/neutron/+bug/1995735 14:54:22 <mmalchuk> yep 14:55:00 <mmalchuk> I've tested all the libs/packages for issue and provide paste links 14:55:35 <mmalchuk> even if we planed to merge https://review.opendev.org/c/openstack/kolla/+/913584 we need fix issue with pyroute2.protocols 14:55:55 <mmalchuk> we install it in the https://review.opendev.org/c/openstack/kolla/+/904781 via packages 14:56:08 <mmalchuk> and it can't be uninstalled via pip 14:58:41 <bbezak> r-krcek: let me get back to you on your services/filter question, I need to check this feature 14:58:49 <SvenKieske> lol? 14:59:07 <SvenKieske> according to https://openstack-ci-reports.ubuntu.com/reports/cloud-archive/yoga_versions.html the package is not called "python3-pyroute2" but "pyroute2" 14:59:25 <SvenKieske> but it's installed via "python3-pyroute2" https://review.opendev.org/c/openstack/kolla/+/904781/3/docker/openstack-base/Dockerfile.j2 14:59:40 <SvenKieske> the other versions in the CI also report this package name afaik 14:59:51 <mmalchuk> there is no issues with packages 15:00:07 <kevko> SvenKieske: pyroute2 is source package name 15:00:19 <mmalchuk> in current images we have pyroute2 on both venv and system 15:00:22 <kevko> SvenKieske: python3-pyroute2 is binary package 15:00:42 <SvenKieske> kevko: ah nice, madness.. 15:00:58 <mmalchuk> I've publeshed an edit for https://review.opendev.org/c/openstack/kolla/+/913584 15:01:20 <SvenKieske> mmalchuk: understood, seems you are right (beside the HWE kernel version, but those are a moving target anyway) 15:01:32 <kevko> SvenKieske: standard 15:02:02 <SvenKieske> I guess we are in overtime now. 15:02:03 <bbezak> ok guys, I need to jump to some call. However please continue the discussion of course :) 15:02:12 <bbezak> thank you all 15:02:14 <bbezak> #endmeeting