13:00:24 <mnasiadka> #startmeeting kolla 13:00:24 <opendevmeet> Meeting started Wed Sep 25 13:00:24 2024 UTC and is due to finish in 60 minutes. The chair is mnasiadka. Information about MeetBot at http://wiki.debian.org/MeetBot. 13:00:24 <opendevmeet> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 13:00:24 <opendevmeet> The meeting name has been set to 'kolla' 13:00:29 <mnasiadka> #topic rollcall 13:00:30 <mnasiadka> o/ 13:00:32 <SvenKieske> o/ 13:00:33 <jovial> \o 13:00:34 <bbezak> o/ 13:00:37 <kevko> \o/ 13:00:43 <mhiner> o/ 13:00:47 <r-krcek> 0/ 13:01:35 <mnasiadka> #topic agenda... (full message at <https://matrix.org/oftc/media/v1/media/download/AfvCuoxbKukDvfMYa1XqelHiI6JA2oy1yrxh5OX-liMapP23Mk5VHpYGNwZDdFVV4TB1g_TTAeUxVm6vFfY2Ur9CeSKv1GuQAG1hdHJpeC5vcmcvSEVTTFBoWldjanVBSWtJT0xKSWlqV0xS>) 13:01:38 <mnasiadka> #topic CI status 13:01:49 <mnasiadka> Green? Red? Amber? Watermelon? 13:02:37 <frickler> o/ 13:02:50 <frickler> that agenda topic was mangled in the matrix 13:02:53 <SvenKieske> what about honeydew melon? :D 13:03:14 <mnasiadka> Yeah, switched to Element, probably it's mangling ;-) 13:03:23 <mnasiadka> I'll try better next time 13:03:25 <SvenKieske> yeah, you need to post agenda line by line for poor old IRC 13:03:28 <mnasiadka> CI looks not bad, except aarch64 13:04:32 <mnasiadka> frickler: do you know if there's any outlook on aarch64 improvements? 13:04:59 <frickler> mnasiadka: none that I'm aware of 13:05:08 <mnasiadka> ok, it is what it is 13:05:16 <mnasiadka> #topic Current cycle planning 13:06:17 <mnasiadka> So, Ubuntu 24.04 seems to be closer, some small CI failures to revisit and I guess I'll ask for reviews next week 13:06:21 <mnasiadka> anything else major that we should be thinking of? 13:06:27 <mnasiadka> There's RMQ 4.0 13:06:37 <mnasiadka> As in no patches, but it got released 13:07:00 <mnasiadka> Since 4.0 release - 3.13 community support has ended 13:07:12 <SvenKieske> yes, we need to shift everything over from the old queues afaik for that to work, so afaik we need to finish the fanout/reply queue stuff 13:07:23 <mnasiadka> Sounds fantastic 13:07:31 <SvenKieske> for that to work in turn, we need the queue manager which kevko is working on 13:07:40 <opendevreview> Michal Arbet proposed openstack/kolla-ansible master: Drop support for py38, py39 https://review.opendev.org/c/openstack/kolla-ansible/+/929408 13:07:40 <opendevreview> Michal Arbet proposed openstack/kolla-ansible master: Add ansible-core as a dependency https://review.opendev.org/c/openstack/kolla-ansible/+/922369 13:07:53 <SvenKieske> but he seems busy :P ;) 13:08:13 <kevko> with queue manager it's little bit hard ..we need to bind /dev/shm to containers using oslo.messaging 13:08:28 <mnasiadka> is that secure? 13:08:45 <SvenKieske> just..bind mount it? :) /dev/shm is actually just a fancy way to say "tmpfs". 13:09:07 <kevko> well, i was thinking about the option to create tmpfs and mount as /dev/shm into container 13:09:14 <SvenKieske> I hope oslo has guarded it and there are no race conditions with concurrent access :D 13:09:20 <kevko> but then I realized we have already some containers which has /dev/shm already mounted 13:09:42 <mnasiadka> Isn't --ipc=host for that? 13:09:44 <kevko> and ...moreover ...in classic openstack installations from packages ..there is shared memory also ... 13:09:56 <kevko> no, ipc turning more than /dev/shm 13:10:28 <kevko> but yes ..if you set ipc_mode ...it's effectively turn on bind /dev/shm into container 13:10:40 <kevko> but we don't need it ... 13:10:44 <mnasiadka> there's the alternative of shared ipc for containers 13:11:06 <mnasiadka> but that was a bit... tricky, because you start a container pointing to other running containers ipc 13:11:09 <frickler> maybe we should stop doing containers if we share everything again anyway in the end? (almost, but not completely non-serious) 13:11:10 <SvenKieske> I think the short term conclusion is we need some time before we can truly merge rmq 4? 13:11:18 <mnasiadka> yeah, that's another point 13:11:39 <mnasiadka> if we need shared mem for all containers, why even containers? 13:12:37 <mnasiadka> so, is that bloody queue manager completely required for RMQ 4? 13:13:30 <SvenKieske> well if we wanna split hairs it's afaik not required for rmq, but for how oslo/openstack uses rmq :) 13:13:44 <kevko> frickler: It is still an effective way to deploy OpenStack services and, for example, run two different versions on one system... it's containerized. Anyway... we use also like net=host namespace...it's similar 13:14:00 <mnasiadka> the oslo.messaging patch says it's optional 13:14:01 <kevko> frickler: But yes, I understand your comment.. :) 13:14:19 <SvenKieske> what it does is essentially keeping track of queue names, so they always stay the same "per host" (that's the tricky part with containers). so consumers can just always bind to the same queue 13:14:58 <kevko> mnasiadka: it's optional yes ... but we also provides ability to override configuration per user ...so if user choose to override and turn on the queue_manager ...he brokes his cloud because we are kolla ..right ? 13:15:08 <SvenKieske> xD 13:15:12 <kevko> and it's also very nice feature :) 13:15:26 <mnasiadka> ok, but it doesn't sound like required 13:15:43 <SvenKieske> well for first release we could put big fat warning that queue manager is not yet supported? if it works? 13:15:46 <kevko> so should we add precheck if user override includes queue_manager = true and fail the reconfigure ? :D 13:16:05 <mnasiadka> user overrides mean user knows what he's doing 13:16:06 <kevko> but agree that it's not important issue for now 13:16:20 <mnasiadka> we never checked what is in user overrides 13:16:28 <kevko> don't forget it's also requirement for streams for fanout 13:16:34 <mnasiadka> so let's not create a drama 13:16:56 <kevko> agree ... 13:17:23 <kevko> it's mainly my downstream project which i am (as always) tracking in gerrit ... and I like that feature 13:18:35 <mnasiadka> Well, if we can create a tmpfs volume and mount that as shm, that would probably be a bit better 13:19:56 <kevko> That's most clean i think ...but i need to test if - when you turn on the ipc if it's not automatically mounted host /dev/shm and overrides your tmpfs mount 13:20:19 <mnasiadka> well, we only run libvirt with ipc=host IIRC 13:20:26 <mnasiadka> that doesn't interact with RMQ 13:20:43 <kevko> hacluster ? 13:20:53 <kevko> cinder ? 13:21:09 <kevko> masakari ? 13:21:12 <mnasiadka> ah, and nova-compute 13:21:16 <kevko> manila ? 13:21:36 <mnasiadka> well then let's drop containers 13:21:37 <mnasiadka> :) 13:21:41 <kevko> we already has bunch of containers host's /dev/shm accessible 13:22:16 <kevko> just saying .. :D 13:22:24 <SvenKieske> various neutron stuff also has quite some host access (necessary for SRIOV etc) afaik 13:22:39 <mnasiadka> Sounds like a lot of work 13:22:49 <mnasiadka> Maybe it's easier to write a container-friendly queue_manager? :) 13:22:52 <SvenKieske> it's just inherent complexity, the k8s deployment tools are basically doing the same thing 13:24:44 <mnasiadka> so then it probably means we need to identify which containers need to have "/dev/shm" mounted 13:24:52 <kevko> Containerization is still great for delivering code; we don't necessarily have to use everything. Moreover, we also copy config files into the container instead of using binds as it should be... for example. 13:24:52 <mnasiadka> and that's it 13:25:38 <kevko> I just want to say that we, as Kolla, definitely don't do everything the way it should be done... and with the complexity of OpenStack, I don't think it's even possible. 13:25:45 <mnasiadka> well, binds have the problem of permissions 13:26:10 <mnasiadka> so probably that's why the project started with this 13:26:38 <mnasiadka> anyway, we have probably other priorities this cycle 13:27:22 <kevko> Anyway, I will work on it and track it in Gerrit. It's probably not worth discussing it much. I’d rather ask about ProxySQL. I’ve fixed everything, including IPv6, and I think it’s OK to switch. 13:27:22 <mnasiadka> MariaDB bump is not happening, unless they release a new version of 11.4 where they fixed SST between old MariaDB and new MariaDB 13:28:52 <mnasiadka> Well, I think I commented we need to remove the unused mariadb-clustercheck running container in upgrade tasks 13:29:11 <mnasiadka> if we're changing the default 13:29:48 <kevko> mnasiadka: yeah, I replied that it can be used by external scripts maybe ... and in reno it's specified that it can be removed manually 13:30:12 <kevko> mnasiadka: but I agree ... i will write something in followup ..so i will add on the top of relation chain .. 13:30:20 <mnasiadka> ok then 13:30:28 <kevko> +1 13:30:57 <mnasiadka> I've added my +2 13:31:11 <mnasiadka> https://review.opendev.org/c/openstack/kolla-ansible/+/913724 13:31:25 <mnasiadka> Do we need to backport any of the fixes you had to do? 13:32:03 <kevko> yep ..we should one 13:32:35 <kevko> this one https://review.opendev.org/c/openstack/kolla-ansible/+/929848 only 13:32:40 <kevko> it's just ipv6 ... 13:33:30 <kevko> nothing more ..because that dance with --initial and removal of db file << ..this was only in master ...lower versions still working fine ... 13:33:55 <kevko> and maybe this ... https://review.opendev.org/c/openstack/kolla-ansible/+/930077/4 << but this sounds like a feature .... 13:34:06 <kevko> mnasiadka: thanks 13:34:36 <opendevreview> Merged openstack/kolla-ansible stable/2024.1: Fix links in docs for unmaintained releases https://review.opendev.org/c/openstack/kolla-ansible/+/928579 13:34:38 <opendevreview> Merged openstack/kolla-ansible stable/2023.2: Fix links in docs for unmaintained releases https://review.opendev.org/c/openstack/kolla-ansible/+/928581 13:35:14 <opendevreview> Merged openstack/kolla-ansible stable/2023.1: Fix links in docs for unmaintained releases https://review.opendev.org/c/openstack/kolla-ansible/+/928587 13:35:14 <mnasiadka> frickler, SvenKieske : around backporting 930077 - it would be useful for my $downstream - are you fine with doing that? 13:35:33 <mnasiadka> doesn't include any inventory changes, so won't include any problems hopefully 13:36:22 <kevko> nice to hear that you also using ProxySQL ;-) 13:36:29 <mnasiadka> Not yet, but planning to. 13:36:35 <kevko> ah, ok 13:36:36 <SvenKieske> we need an exception for those, no? via ML? but looks okayish I think 13:36:38 <mnasiadka> And monitoring is an important aspect 13:37:15 <mnasiadka> kevko: propose the backports please for the ipv6 and the prometheus endpoint via Gerrit and we'll sort it then 13:37:16 <kevko> ML ? what is the shortcut for ? :D 13:37:21 <mnasiadka> mailing list 13:37:25 <kevko> ah ok 13:37:26 <mnasiadka> but no, never did it that way 13:37:28 <kevko> mnasiadka: ack 13:37:52 <mnasiadka> #topic Additional agenda (from whiteboard) 13:38:13 <mnasiadka> mhiner - please review: Move to high level client in DockerWorker https://review.opendev.org/c/openstack/kolla-ansible/+/908295 13:38:19 <opendevreview> Grzegorz Koper proposed openstack/kolla-ansible stable/2023.1: Fixing typo in etc/kolla/globals.yml https://review.opendev.org/c/openstack/kolla-ansible/+/930466 13:39:25 <mnasiadka> mhiner: commented, will also do a proper review 13:39:34 <mhiner> thanks 13:39:40 <opendevreview> Grzegorz Koper proposed openstack/kolla-ansible stable/2023.2: Fixing typo in etc/kolla/globals.yml https://review.opendev.org/c/openstack/kolla-ansible/+/930467 13:39:52 <mnasiadka> r-krcek: https://review.opendev.org/c/openstack/kolla-ansible/+/923110 13:40:04 <opendevreview> Grzegorz Koper proposed openstack/kolla-ansible stable/2024.1: Fixing typo in etc/kolla/globals.yml https://review.opendev.org/c/openstack/kolla-ansible/+/930468 13:41:16 <mnasiadka> ok, commented - would be nice to get a python based CLI this cycle I guess 13:42:12 <mnasiadka> SvenKieske: I've seen you did some reviews - can you help r-krcek to get those over the line? 13:42:40 <mnasiadka> Another one from r-krcek - https://review.opendev.org/c/openstack/kolla-ansible/+/599735 13:43:38 <mnasiadka> reviewed 13:43:56 <SvenKieske> yeah, probably over the next days, still stuck in upgrading stuff, sadly have considerable less time currently during working hours for reviews :-/ 13:44:15 <SvenKieske> will make some room for reviews in my free time I guess 13:44:53 <mnasiadka> Last one is a question if anybody is interested in Swift and would like to help defining tests in https://review.opendev.org/c/openstack/kolla-ansible/+/797498 13:46:23 <mnasiadka> If yes, wuchunyang and r-krcek would be probably grateful 13:46:25 <mnasiadka> #topic Open discussion 13:46:30 <mnasiadka> anything else to cover today? 13:47:16 <r-krcek> Thank you guys for the reviews and comments :) 13:48:31 <kevko> i would like to ask again for some reviews ...but you know which one right :D 13:49:04 <mnasiadka> r-krcek: if you are interested in being a more permanent part of Kolla community - given your not small contributions - you could review other people patches and become a core some day 13:49:25 <mnasiadka> kevko: yeah, will have a look on your patches tomorrow 13:50:15 * frickler will be on holiday for the coming three weeks, just fyi. will be back in time for the ptg 13:50:39 <mnasiadka> Remember PTG for Kolla starts on Tue - I already booked the slots 13:50:46 <frickler> ack 13:50:48 <mnasiadka> #link https://etherpad.opendev.org/p/kolla-epoxy-ptg 13:50:48 <kevko> ack 13:50:54 <mnasiadka> Please try to populate the topics 13:50:59 <mnasiadka> we have 2x3 hours 13:51:08 <mnasiadka> (and there's kayobe as well) 13:51:10 <mnasiadka> jovial: ^^ 13:51:38 <mnasiadka> Ok then, that's all for me - see you next week (have fun frickler !) 13:51:55 <jovial> awesome - I'll populate a few topics. Would definitely appreciate any input there. 13:51:59 <SvenKieske> see you :) 13:52:15 <mmalchuk> o/ 13:52:20 <mnasiadka> #endmeeting