13:03:36 <mnasiadka> #startmeeting kolla 13:03:36 <opendevmeet> Meeting started Wed Jul 3 13:03:36 2024 UTC and is due to finish in 60 minutes. The chair is mnasiadka. Information about MeetBot at http://wiki.debian.org/MeetBot. 13:03:36 <opendevmeet> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 13:03:37 <opendevmeet> The meeting name has been set to 'kolla' 13:03:39 <mnasiadka> #topic rollcall 13:03:40 <mnasiadka> o/ 13:03:48 <mmalchuk> щ/ 13:03:50 <jovial> o/ 13:03:51 <mmalchuk> щ/ 13:03:55 <mmalchuk> o/ 13:03:58 <mhiner> o/ 13:03:59 <r-krcek> \0/ 13:04:05 <frickler> \o 13:04:47 <ihalomi> \o 13:05:44 <mnasiadka> #topic agenda 13:05:44 <mnasiadka> * CI status 13:05:44 <mnasiadka> * Release tasks 13:05:44 <mnasiadka> * Regular stable releases (first meeting in a month) 13:05:44 <mnasiadka> * Current cycle planning 13:05:46 <mnasiadka> * Additional agenda (from whiteboard) 13:05:46 <mnasiadka> * Open discussion 13:05:48 <mnasiadka> #topic CI status 13:06:01 <mnasiadka> overall green, unless I stopped understanding colors ;-) 13:07:30 <mnasiadka> debian-aarch64 kolla periodics failed, but I guess that's something temporary 13:07:36 <mnasiadka> #topic Release tasks 13:07:40 <mnasiadka> none for this week 13:07:51 <mnasiadka> #topic Current cycle planning 13:08:02 <frickler> what about stable releases? 13:08:03 <mnasiadka> I've re-started work on bumping Ansible 13:08:08 <mnasiadka> Ah, stable releases - sorry 13:08:15 <mnasiadka> #topic Regular stable releases (first meeting in a month) 13:08:26 <mnasiadka> Any volunteer to raise patches for stable releases? 13:08:32 <mnasiadka> (it's been a while since we had those) 13:08:47 <frickler> I can propose a patch unless someone wants to up their stats 13:10:25 <mnasiadka> well, it would be good it's not you - because then it's awkward if you +2 the releases patch 13:10:34 <mnasiadka> and it seems the release team numbers are not so great ;-) 13:11:26 <frickler> I don't mind doing awkward things ;) but feel free to propose the patch yourself 13:11:40 <mnasiadka> yeah, seems like it 13:11:42 <mnasiadka> I'll do it 13:11:45 <mmalchuk> frickler show me a patch I'll push it ;) to save from awkward things ;) 13:11:57 <mnasiadka> #topic Current cycle planning (again) 13:12:00 <frickler> there is some doc on how to do it 13:12:07 <mnasiadka> mmalchuk: it's in kolla release docs 13:12:27 <mnasiadka> So, continuing 13:12:36 <mmalchuk> if no hurry will do tomorow or at night 13:12:39 <mnasiadka> ansible-core 2.16 (and 2.17) require controller to be python3.10+ 13:12:46 <mnasiadka> so we need to drop OpenEuler 13:13:01 <frickler> mmalchuk: https://docs.openstack.org/kolla/latest/contributor/release-management.html#maintained (needs an update in the active stable branches list) 13:13:01 <mnasiadka> we need to use python3.12 in Rocky 9 CI (only on the ansible controller) 13:13:02 <mmalchuk> finally) 13:13:16 <mnasiadka> so we'll also need to document this 13:13:23 <mmalchuk> frickler thnaks 13:13:30 <mmalchuk> *thanks 13:13:37 <mnasiadka> I'm working on getting the patch green - currently some hashicorp vault stuff is failing and zun 13:13:41 <frickler> but >= py3.10 should still be fine? 13:13:55 <mnasiadka> yes, Debian and Ubuntu is fine 13:14:05 <mnasiadka> and RHEL9 (so Rocky 9) has rpm packages for python3.12 13:14:15 <mnasiadka> so we're fine 13:14:26 <mnasiadka> and new Ubuntu will get something like 3.12 or later 13:14:43 <jovial> mnasiadka, Just thinking about the bump in kayobe. Will we have the usual issue with selinux/dnf bindings? 13:14:59 <mnasiadka> jovial: on the host ansible manages you can even have 3.6 13:15:07 <mnasiadka> so no issues with dnf bindings I guess 13:15:13 <mnasiadka> selinux I mean 13:15:20 <mnasiadka> hopefully 13:15:21 <jovial> But on the control host? 13:15:34 <mnasiadka> we'll see I guess 13:15:41 <jovial> I think we install a few packages 13:15:50 <mnasiadka> then we need to start working on it as well ;) 13:16:16 <jovial> Sounds like a plan 13:18:31 <jovial> I will knock up the kayobe change that depends on your patch :) 13:18:39 <mnasiadka> great, thanks 13:18:53 <mnasiadka> #topic Additional agenda (from whiteboard) 13:19:21 <mnasiadka> (mhiner)[July, 3rd]: Please review(Zuul is now happy): https://review.opendev.org/c/openstack/kolla-ansible/+/836941 13:21:39 <mnasiadka> mhiner: there are some unresolved comments - can you fix that? 13:21:59 <mnasiadka> I'll have a look in that patch later today 13:22:03 <mnasiadka> (SvenKieske)[June, 28th] I'd like some reviews and possibly RP+1 on these, please: 13:22:03 <mnasiadka> https://review.opendev.org/c/openstack/kolla-ansible/+/915403 (harden haproxy TLS config) 13:22:03 <mnasiadka> https://review.opendev.org/c/openstack/kolla-ansible/+/922840 (add a config validation check to haproxy) 13:22:03 <mnasiadka> https://review.opendev.org/c/openstack/kolla-ansible/+/921371 (sec hardening horizon by not mounting host tmp) 13:22:03 <mnasiadka> https://review.opendev.org/c/openstack/kolla-ansible/+/907977 (rmq precheck: also check fanout and reply queues) 13:24:48 <mhiner> mnasiadka: will add some replies, but in the end, Sven has to decide if he is happy with them. 13:25:08 <mnasiadka> mhiner: I mean it's easier for next reviewer if there are resolved comments :) 13:26:22 <mnasiadka> ok, went through Sven's patches, added -1 to each of them ;-) 13:26:34 <mnasiadka> (ihalomi)[3rd July] docker_worker refactor is now passing tests and is ready to review 13:26:34 <mnasiadka> https://review.opendev.org/c/openstack/kolla-ansible/+/908295 13:26:34 <mnasiadka> also refactor of container_facts is ready to be reviewed by others 13:26:34 <mnasiadka> https://review.opendev.org/c/openstack/kolla-ansible/+/911417 13:26:34 <mnasiadka> simple fix for bug found by kevko: 13:26:36 <mnasiadka> https://review.opendev.org/c/openstack/kolla-ansible/+/923231 13:26:43 <mnasiadka> ok then, let me have a look 13:27:34 <mnasiadka> first one is big, so RP+1 for later review 13:28:02 <mnasiadka> second one the same 13:28:47 <mnasiadka> and third needs a bug 13:29:07 <mnasiadka> jovial(3rd July) 13:29:07 <mnasiadka> Should we add a kayobe job to kolla CI? 13:29:07 <mnasiadka> Circular dependency so would need to be non-voting (I think) 13:29:07 <mnasiadka> Based on discussion in https://review.opendev.org/c/openstack/kolla-ansible/+/920368 13:29:10 <mnasiadka> ok then 13:29:45 <mnasiadka> jovial: how long is the job running? is it way longer than the regular aio kolla-ansible jobs? 13:30:37 <mnasiadka> just propose adding something, but I would prefer it's two jobs, not more :) so maybe aio rocky/ubuntu 13:30:37 <jovial> ~ 1h 08m 59s 13:30:48 <mnasiadka> and non-voting for starter 13:31:08 <frickler> are you talking about kolla really or k-a? 13:31:16 <mnasiadka> I think k-a 13:31:17 <ihalomi> mnasiadka: what do you mean it needs a bug? 13:31:21 <jovial> Yeah, I think it would be hard to make it voting as quite often we need a fix in kolla-ansible to get kayobe-ci to work 13:31:21 <mnasiadka> in kolla it doesn't make any sense 13:31:31 <mnasiadka> unless we have kayobe image build jobs, which we probably don't :) 13:31:55 <mnasiadka> jovial: well, it would make us merge things that don't break kayobe e.g. are backwards compatible :) 13:33:14 <jovial> True: might lead to use having to disable to get stuff into k-a occasionally (if its something that is external to k-a that has broken us) 13:33:41 <mnasiadka> ok, so adding two kayobe aio jobs into kolla-ansible - that makes sense 13:33:56 <mnasiadka> regarding frickler's earlier question :) 13:34:07 <jovial> I just wanted to check that it wouldn't get immediately rejected if I proposed something :) 13:34:41 <mnasiadka> we don't immediately reject anything, you can get grilled and asked what are you doing - but not an immediate -2 ;-) 13:35:23 <jovial> For kolla: We do now build the container images: https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_abc/922795/5/check/kayobe-seed-images-rocky9/abcc463/primary/ansible/overcloud-container-image-build 13:35:55 <mnasiadka> Maybe that's a second step, but don't know how would we be able to break kayobe image build :) 13:36:05 <jovial> Just what I was thinking :D 13:36:48 <mnasiadka> ok then, that's it from the whiteboard 13:37:01 <mnasiadka> #topic Open discussion 13:37:04 <mnasiadka> Anybody, anything? 13:37:09 <r-krcek> Hi! I am thinking of moving kolla-asible CLI into python. m.nasiadka recommended to use cliff package - like is used in kayobe-cli or openstack-cli. After reading the docs I noticed that the command structure is different from kolla-ansible. (the commands have subcommands eg. kayobe baremetal compute manage) Would you be more inclined if the new CLI would be are similar to the old one or if it followed the convention used in kayobe-cli and op 13:37:09 <jovial> OK, so I'll try and submit a patch to add the AIO job when I get a bit of time :) 13:38:42 <jovial> How many people use kolla-ansible CLI? 13:38:43 <frickler> r-krcek: seems your message was cut off after "kayobe-cli and op" 13:39:08 <r-krcek> continuation: For example "kolla-ansible bifrost deploy" instead of "kolla-asnible deploy-bifrost" or "kolla-ansible mariadb backup" and "kolla-ansible mariadb recovery" "kolla-ansible containers deploy" and "kolla-ansible containers stop" etc. Plus some features like multiple environments and hooks are marked as experimental. Do you think it is worth it/wanted to also add these features? What do you think? 13:39:42 <jovial> Oh my bad, was thinking of https://opendev.org/openstack/kolla-cli 13:39:46 <mnederlof> doesn´t everyone use the kolla-ansible cli? 13:40:07 <mnederlof> or is it recommended to implement the roles in your own playbooks? 13:40:48 <mnasiadka> Well, from code perspective - I would prefer we have similar code for CLI - so it's easier to maintain as a project 13:40:55 <jovial> mnederlof, Think this was confusion on my part, this is about writing the kolla-ansible bash script in python I guess 13:41:07 <mnasiadka> yes, that's about rewriting bash script to python 13:41:14 <mnasiadka> and if we have something working for Kayobe 13:41:19 <mnasiadka> then for me it's a no brainer task 13:41:23 <mnasiadka> or is it just me? :) 13:42:23 <mnederlof> i am fine with migrating it to python, what comes to mind is that it might be harder to have live output while ansible runs 13:42:43 <mnasiadka> well, we have that working in Kayobe 13:42:47 <mnederlof> awesome 13:42:53 <mnasiadka> so let's not reinvent the whell 13:42:55 <mnasiadka> wheel 13:42:58 <mnasiadka> r-krcek: what do you say? 13:43:16 <mnasiadka> r-krcek: I wouldn't add any features now - just achieve 1:1 feature parity 13:43:33 <jovial> +1 13:43:34 <mnederlof> as a first release anyway :) 13:43:37 <mnasiadka> but in the past kolla-cli also supported multiple environments (but not in the same way as Kayobe does that today) 13:43:44 <r-krcek> I agree qith m.nasiadka. Lets not reinvent the wheel. :) 13:44:14 <mnederlof> i also have another question, regarding the use of systemd for starting/stopping the docker containers, does anyone know why this was changes? 13:44:16 <mnederlof> *changed 13:44:24 <mnasiadka> because we also support podman 13:44:29 <mnasiadka> and that was the only sensible way 13:44:35 <mnederlof> ah, ok 13:44:41 <r-krcek> Okay, so for now just as close as I can get it to 1:1 with the old interface. Thanks for the input 13:45:55 <mnederlof> we seem to have some issues with systemd, when a container is stopped continuesly (for example a compute node that is restarted, and does not have its compute_id file yet and is still known in the nova database) 13:46:11 <mnasiadka> I think we have a bug around compute_id and managing that properly 13:46:18 <mnasiadka> currently we leave that to Nova 13:46:28 <mnasiadka> and compute_id file gets put in /var/lib/nova 13:46:38 <mnederlof> yes, in the docker volume 13:46:52 <mnasiadka> well, docker volume should be there without any problems 13:46:56 <mnederlof> yes 13:47:08 <mnasiadka> I was thinking that some people put that on NFS - and if that's not mounted or unavailable you can get into issues 13:47:50 <jovial> I guess you could use add a dependency to docker to not start until the storage is mounted for that case? 13:47:52 <mnasiadka> Have you raised a bug and attached some logs? 13:48:09 <mnederlof> the compute_id file is not really the problem though, a restarting container i can fix, but having to reboot the server because systemd is on the fence is kind of annoying (also because the whole system becomes sluggish) 13:48:28 <mnederlof> no not yet, as i am not sure what the trigger is yet 13:48:31 <mnederlof> will do that though 13:48:35 <mnasiadka> well, if we can find the cause for that behaviour - maybe we can fix that 13:48:46 <mnederlof> but was thinking about making systemd optional 13:48:46 <mnasiadka> feel free to raise a bug and leave the link here - so we can have a look 13:49:03 <mnederlof> when using docker 13:49:36 <mnasiadka> well, that's sort of a step back - we haven't observed any issues with systemd approach yet 13:49:48 <mnasiadka> so raise that bug and we can continue in there :) 13:49:53 <mnederlof> yes, will do that :) 13:50:01 <mnasiadka> thanks 13:50:07 <mnasiadka> ok, I think that's it for today 13:50:13 <mnasiadka> Thank you all for coming - see you next week! 13:50:15 <mnasiadka> #endmeeting