13:00:24 <bbezak> #startmeeting kolla 13:00:24 <opendevmeet> Meeting started Wed Jun 14 13:00:24 2023 UTC and is due to finish in 60 minutes. The chair is bbezak. Information about MeetBot at http://wiki.debian.org/MeetBot. 13:00:24 <opendevmeet> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 13:00:24 <opendevmeet> The meeting name has been set to 'kolla' 13:00:31 <bbezak> #topic rollcall 13:00:37 <mmalchuk> \o 13:00:39 <mattcrees> o/ 13:00:47 <SvenKieske> o/ 13:00:55 <kevko> gre\o/ 13:00:56 <frickler> and /me is surprised, I'd assumed we'd skip this one, too. but well, I'm here 13:01:34 <bbezak> I recon we can chat about pressing issues, even when some of us are in vancouver sleeping :) 13:01:42 <mmalchuk> frickler only mnasiadka skip this meeting as he said 13:01:53 <hrw> \~o~\ 13:02:44 <bbezak> #topic agenda 13:02:46 <bbezak> * Roll-call 13:02:46 <bbezak> * Agenda 13:02:46 <bbezak> * Announcements 13:02:46 <bbezak> * Review action items from the last meeting 13:02:46 <bbezak> * CI status 13:02:47 <bbezak> * Release tasks 13:02:47 <bbezak> * Regular stable releases (first meeting in a month) 13:02:49 <bbezak> * Current cycle planning 13:02:49 <bbezak> * Additional agenda (from whiteboard) 13:02:51 <bbezak> * Open discussion 13:04:01 <bbezak> I don't have much too say about announcements and review actions items from the last meeting 13:04:25 <bbezak> let's go to the CI status then 13:04:33 <bbezak> #topic CI status 13:04:41 <mmalchuk> red 13:05:10 <mmalchuk> rabbitmq issue the main thing 13:05:22 <hrw> just do upgrade to 3.12/26 13:05:28 <mmalchuk> +1 13:05:46 <frickler> 3.12.0? very courageous 13:05:48 <SvenKieske> as said above, there seem to be new IO issues, see e.g. https://zuul.opendev.org/t/openstack/build/e804ca9b01f24a4ca7e1d7edb3e96fb3/log/job-output.txt 13:05:54 <mmalchuk> frickler why? 13:05:59 <frickler> and not very feasible for stable branches 13:06:10 <frickler> because .0 is like "alpha" release 13:06:22 <mmalchuk> tommorow it would be .1 13:06:24 <hrw> frickler: that's why master first? 13:06:42 <mmalchuk> lets pin .* 13:07:30 <hrw> rabbitmq/erlang was always a problem 13:07:37 <frickler> so maybe for master, ok, but that doesn't handle stable branches and upgrade tests 13:08:01 <bbezak> yeah, surely stable branches cannot go with 3.12.0 13:08:04 <mmalchuk> as mnasiadka said we a firefighters) so wi will fix it again and again 13:08:20 <kevko> Arnaud Cogoluègnes replied regarding rabbitmq 13:08:35 <hrw> I do not have hours assigned to kolla anymore so will ignore rmq this time 13:08:52 <hrw> kevko: url? 13:08:56 <SvenKieske> sure, there are, especially for upgrades, quite some preconditions mentioned in the 3.12 release notes, before you can upgrade from 3.11; don't know if we meet them 13:09:03 <frickler> probably in moderation still 13:09:18 <kevko> it is not in mail archive ..but check openstack discuss ..he is suggesting to switch to cloudsmith ..i am preparing email that we already replaced this repo 13:09:59 <frickler> so what about arm then? 13:10:15 <mmalchuk> postpone ? 13:10:16 <hrw> sometimes I wonder when kolla team get tired enough and remove whole aarch64 support and then start cutting distros 13:10:33 <mmalchuk> fix x86_64 first? 13:10:36 <bbezak> SvenKieske: those IO issues looks like zuul issues, maybe it is a case for #opendev channel ? 13:10:42 <kevko> hrw: +1 :D 13:10:57 <mmalchuk> +1 lol 13:11:05 <SvenKieske> bbezak: yeah, was wondering the same; I'll post them there 13:11:21 <bbezak> SvenKieske: thx 13:13:31 * SvenKieske has the impression that there simply does not exist a stable, available mirror for all past erlang/rmq releases, regardless the architecture. 13:14:01 <bbezak> that would a bummer :) 13:14:05 <mmalchuk> it exists but lack of aarch64 13:14:07 <bbezak> and a bit crazy 13:14:09 <hrw> SvenKieske: exactly 13:15:01 <mmalchuk> #opendev sleeps with vancouver) 13:15:02 <SvenKieske> mmalchuk: I dunno, I mean we get the suggestion to use cloudsmith, but we just disabled that because it was broken, no? where is that magic mirror that doesn't remove past patch releases ? 13:15:49 <opendevreview> Michal Arbet proposed openstack/kolla master: Bump proxysql version https://review.opendev.org/c/openstack/kolla/+/886093 13:16:13 <mmalchuk> SvenKieske we have many suggestions but not choose right) 13:17:06 <hrw> SvenKieske: we have 3 distros and two architectures. and 5-6 distro versions. 13:17:22 <frickler> so the only solution: build erlang+rmq from source within a container? 13:17:30 <hrw> SvenKieske: covering it for rmq/erlang is painful 13:17:40 <SvenKieske> so, fix x86_64 first, or what? fine with me; afaik I don't know of any arm prod systems, but that doesn't mean a lot. 13:17:47 <kevko> https://review.opendev.org/c/openstack/kolla/+/885857 13:17:48 <SvenKieske> hrw: sure; I know :) 13:18:02 <mmalchuk> frickler built from source? kidding? 13:18:04 * hrw off - other duties 13:18:10 <bbezak> somebody contacted ppa admins? 13:18:13 <kevko> me 13:18:17 <bbezak> thx hrw 13:18:20 <SvenKieske> frickler: that sounds at least somewhat viable, though I never build anything in the erlang space. 13:18:42 <kevko> but mail not archived because of message size ... probably my picture of company in footer ? :/ 13:19:06 <mmalchuk> lets drop the arm and one problem solved 13:20:03 <SvenKieske> the less nuclear option maybe would be to contact ppa admins and ask if we can help maintain that. as I understood it was "just" a matter of a package being dropped. co maintaining is certainly less a burden than rebuilding everything ourselves? 13:20:13 <kevko> https://paste.openstack.org/show/bUaFtanAO13OvMm8O79Q/ << communication 13:20:25 <bbezak> pretty big decision for that meeting I'd say :), let's not rush it and drop support for now 13:20:36 <kevko> SvenKieske do you have arm64 hosts to build ? 13:21:15 <bbezak> so according to them everything is fine? kevko 13:21:21 <bbezak> "network glitch" 13:21:36 <bbezak> https://cloudsmith.io/~rabbitmq/repos/rabbitmq-erlang/packages/?q=distribution%3Aubuntu+AND+distribution%3Ajammy+AND+version%3A1%3A25*+AND+name%3A%27%5Eerlang-base%24%27 13:22:10 <SvenKieske> I mean when I look at the link posted by bbezak everything looks fine? did someone retry the download? 13:22:32 <SvenKieske> maybe that stuff was affected by the aws us-east-1 outage yesterday(?) 13:22:40 <SvenKieske> dunno where cloudsmith hosts 13:22:52 <kevko> we are using ppa.launchpad.net because armd64 ...but they removed 25* ..so there is only 26* ... when you switch to cloudsmith ...it's ok ...but no support for arm64 13:23:27 <SvenKieske> kevko: ah thanks for correcting me; I mixed those repos up (we switch too often :D ) 13:23:47 <mmalchuk> lets vote for temp swtitch to cloudsmith? until we fix arm 13:24:16 <SvenKieske> sounds reasonable from my newbie perspective *shrugs* 13:24:46 <kevko> Launchpad PPA only provides the most recent release of Erlang 25.3.x. Cloudsmith and its mirror currently provides the most recent patch release in the following Erlang series: 13:25:10 <kevko> https://www.rabbitmq.com/install-debian.html#apt-launchpad-erlang 13:25:57 <kevko> we definitely remove and replace ppa repository 13:26:08 <kevko> because it looks like ppa.launchpad.net only maintain latest erlang versions 13:26:09 <SvenKieske> "The Cloudsmith repository has a monthly traffic quota that can be exhausted" 13:26:18 <SvenKieske> https://www.rabbitmq.com/install-debian.html#apt-cloudsmith 13:26:43 <mmalchuk> IMHO Cloudsmith is the best place. then we can talk how to build/sync arm repos and add it later 13:26:48 <kevko> that's the reason why i used the mirros ... 13:26:49 <kevko> The Cloudsmith repository has a monthly traffic quota that can be exhausted. For this reason, examples below use a Cloudsmith repository mirror. All packages in the mirror repository are signed using the same signing key. 13:27:02 <kevko> in my patch 13:27:16 <kevko> ppa1.novemberain.com 13:27:19 <SvenKieske> yeah, sounds also reasonable 13:27:24 <kevko> https://review.opendev.org/c/openstack/kolla/+/885857 13:27:54 <mmalchuk> +1 13:28:47 <kevko> The logs above mentioned both Cloudsmith and PPA, use one or the other, not both. 13:28:47 <kevko> PPA makes only the latest built version available, in this case, 26.0. This is something we don't control with PPA. 13:28:53 <kevko> ^^ he replied 6 minutes ago 13:28:55 <mmalchuk> no matter wich repo, I mean Cloudsmith - lest be a mirror 13:29:29 <bbezak> so cloudsmith looks better in that regard surely 13:30:27 <kevko> yep 13:31:22 <bbezak> tough case indeed. In any case I would rather have fixed stable branches for x86 for now. master can use 26. On the other hand divergence between master and stable branches is not wise in the longer run 13:32:21 <mmalchuk> as I understand we don't want 26 13:32:45 <mmalchuk> even master use 25 from cloudsmith mirror 13:33:00 <bbezak> not in stable branches, until it is proven to be stable and working upgrade 13:33:56 <kevko> we should use cloudsmith for amd64 and something different for amr64 13:34:14 <kevko> what about this http://binaries.erlang-solutions.com/debian/dists/bullseye/contrib/binary-arm64/Packages 13:34:14 <bbezak> looks like it is a way to go 13:34:25 <kevko> this should work 13:35:44 <bbezak> no jammy arm there? 13:35:53 <kevko> nope 13:36:13 <mmalchuk> debian only 13:36:45 <mmalchuk> http://binaries.erlang-solutions.com/ubuntu/dists/jammy/ 13:36:53 <mmalchuk> empty 13:37:21 <bbezak> debian: 13:37:21 <bbezak> erlang: 13:37:21 <bbezak> url: "https://ppa.launchpadcontent.net/rabbitmq/rabbitmq-erlang/ubuntu" 13:37:21 <bbezak> suite: "focal" 13:37:21 <bbezak> component: "main" 13:37:21 <bbezak> gpg_key: "erlang-ppa.gpg" 13:37:29 <bbezak> that is current setting 13:37:29 <SvenKieske> maybe the debian packages "Just work" on jammy? (who am I kidding, but one can hope) 13:37:31 <kevko> hmm we can use debian repo ...but worse is that yammy is based on bookworm if i am correct and bookworm is missing 13:37:34 <bbezak> exactly my point 13:37:35 <bbezak> :) 13:37:46 <bbezak> can you test it kevko ? 13:37:54 <bbezak> in change that is 13:38:37 <kevko> in past hrw built and hosted erlang on his server ..but he already dropped ... 13:38:44 <kevko> maybe we can ask him to build again ? :P 13:39:13 <kevko> well, i don't have arm host ..so i can't :D 13:39:21 <kevko> *can't test 13:39:22 <bbezak> I mean those erlang-solutions.com 13:39:26 <bbezak> binaries 13:39:31 <kevko> i don't have arm64 13:39:52 <bbezak> well I guess in CI it is fine 13:40:04 <kevko> which job is for arm64 ? 13:41:17 <bbezak> kolla-build-ubuntu-aarch64 13:42:03 <bbezak> kolla-build-debian-aarch64, 13:42:07 <bbezak> and then kolla-ansible-debian-aarch64 13:42:13 <bbezak> it has failed in your change 13:42:14 <kevko> https://paste.openstack.org/show/b3mRNUqQi2BVfjRTUhs9/ 13:42:25 <bbezak> yeap 13:42:25 <kevko> yeah i know :D 13:42:28 <bbezak> :) 13:44:17 <bbezak> ok I think we have something to test on. Use stable 25 1.11 from Cloudsmith for amd64, and try maybe those erlang-solutions for arm in the meantime if possible 13:44:54 <bbezak> ok, let's move on. we can continue this discussion on IRC and in the change 13:45:15 <bbezak> if that's ok for you 13:45:37 <frickler> +1 13:45:41 <SvenKieske> +1 13:45:43 <bbezak> #topic Release tasks 13:46:34 <bbezak> we need to do switchover master branch from antelope, I saw some changes from hrw. I'll try to look into that later this week if nobody have a time to do so 13:47:15 <mmalchuk> lets merge https://review.opendev.org/c/openstack/kolla/+/886087 13:47:19 <mmalchuk> it has +2 13:48:05 <mmalchuk> no matter that CI failed so) 13:48:31 <kevko> well, i thought that it will pass and second +2 is needed 13:48:34 <frickler> what about the rocky failure? is that also from rmq repo? 13:48:35 <kevko> i will reset my voe 13:49:00 <bbezak> it is bifrost 13:49:05 <bbezak> that failed in rocky 13:49:19 <kevko> frickler: rocky repo should use different repo for erlang ..but didn't check ..ppa is used only for debubtu 13:49:32 <frickler> so more CI issues 13:49:44 <bbezak> hurray 13:50:12 <frickler> epel install failure 13:50:18 <frickler> https://ef634d1e9a5ddf8bf39c-29c48a4a553d1a0c965532e3378dd6a0.ssl.cf1.rackcdn.com/886087/1/check/kolla-build-rocky9/17bbc7d/kolla/build/000_FAILED_bifrost-base.log 13:50:23 <frickler> might just be unstable? 13:50:46 <bbezak> it might, gpg error 13:51:23 <bbezak> let's recheck it then, let's see 13:51:55 <bbezak> ok, let's go further 13:52:04 <bbezak> #topic Current cycle planning 13:52:44 <bbezak> I assume bookworm, that hrw is working on 13:54:02 <bbezak> anyone has a case to discuss within this topic? 13:54:27 <kevko> nope, i will follow hrw changes and will help if it will be needed 13:54:33 <bbezak> thx kevko 13:54:39 <bbezak> #topic Open discussion 13:54:56 <mmalchuk> please review and merge backports: https://review.opendev.org/q/topic:fix-cyborg-service 13:55:11 <bbezak> will do mmalchuk 13:55:13 <mmalchuk> and there lack of Kayobe reviews again 13:55:25 <mmalchuk> https://review.opendev.org/c/openstack/kayobe/+/879554?usp=search 13:55:30 <mmalchuk> https://review.opendev.org/c/openstack/kayobe/+/861397?usp=search 13:55:40 <mmalchuk> please these two at least 13:56:14 <bbezak> will try to look into that 13:56:33 <kevko> bbezak i've added you as reviewer for my patch ..can u check ? https://review.opendev.org/c/openstack/kolla-ansible/+/877413 13:56:54 <bbezak> sure 13:57:26 <mmalchuk> regarding https://bugs.launchpad.net/kolla-ansible/+bug/1965111 13:57:37 <mmalchuk> should we backport? 13:57:55 <mmalchuk> y, x, w 13:58:34 <mmalchuk> IMHO we need to 13:58:49 <bbezak> if it's not breaking anything I don't think it is an issue 13:59:02 <mmalchuk> maybe you right 13:59:23 <kevko> i think this is not breaking anything 13:59:30 <mmalchuk> but it can be cacouse service_name==service_type in most cases 13:59:47 <mmalchuk> but if we face the issue like Cyborg? 13:59:57 <SvenKieske> I can maybe at least review some of that stuff, as we either way wanted to add more service token stuff in light of the last cve. 14:00:29 <mmalchuk> service tokens - the different problem 14:01:02 <mmalchuk> this is about access rules 14:01:21 <opendevreview> Michal Arbet proposed openstack/kolla master: Bump proxysql version https://review.opendev.org/c/openstack/kolla/+/886093 14:01:24 <SvenKieske> ah okay, I just skimmed it 14:02:04 <bbezak> if you feel that it is a big problem just propose backports, if it is not breaking current deployments 14:02:53 <bbezak> ok I think we can conclude for today, thx for joining and good talk 14:02:55 <bbezak> #endmeeting