13:03:36 <mnasiadka> #startmeeting kolla
13:03:36 <opendevmeet> Meeting started Wed Jul  3 13:03:36 2024 UTC and is due to finish in 60 minutes.  The chair is mnasiadka. Information about MeetBot at http://wiki.debian.org/MeetBot.
13:03:36 <opendevmeet> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
13:03:37 <opendevmeet> The meeting name has been set to 'kolla'
13:03:39 <mnasiadka> #topic rollcall
13:03:40 <mnasiadka> o/
13:03:48 <mmalchuk> щ/
13:03:50 <jovial> o/
13:03:51 <mmalchuk> щ/
13:03:55 <mmalchuk> o/
13:03:58 <mhiner> o/
13:03:59 <r-krcek> \0/
13:04:05 <frickler> \o
13:04:47 <ihalomi> \o
13:05:44 <mnasiadka> #topic agenda
13:05:44 <mnasiadka> * CI status
13:05:44 <mnasiadka> * Release tasks
13:05:44 <mnasiadka> * Regular stable releases (first meeting in a month)
13:05:44 <mnasiadka> * Current cycle planning
13:05:46 <mnasiadka> * Additional agenda (from whiteboard)
13:05:46 <mnasiadka> * Open discussion
13:05:48 <mnasiadka> #topic CI status
13:06:01 <mnasiadka> overall green, unless I stopped understanding colors ;-)
13:07:30 <mnasiadka> debian-aarch64 kolla periodics failed, but I guess that's something temporary
13:07:36 <mnasiadka> #topic Release tasks
13:07:40 <mnasiadka> none for this week
13:07:51 <mnasiadka> #topic Current cycle planning
13:08:02 <frickler> what about stable releases?
13:08:03 <mnasiadka> I've re-started work on bumping Ansible
13:08:08 <mnasiadka> Ah, stable releases - sorry
13:08:15 <mnasiadka> #topic Regular stable releases (first meeting in a month)
13:08:26 <mnasiadka> Any volunteer to raise patches for stable releases?
13:08:32 <mnasiadka> (it's been a while since we had those)
13:08:47 <frickler> I can propose a patch unless someone wants to up their stats
13:10:25 <mnasiadka> well, it would be good it's not you - because then it's awkward if you +2 the releases patch
13:10:34 <mnasiadka> and it seems the release team numbers are not so great ;-)
13:11:26 <frickler> I don't mind doing awkward things ;) but feel free to propose the patch yourself
13:11:40 <mnasiadka> yeah, seems like it
13:11:42 <mnasiadka> I'll do it
13:11:45 <mmalchuk> frickler show me a patch I'll push it ;) to save from awkward things ;)
13:11:57 <mnasiadka> #topic Current cycle planning (again)
13:12:00 <frickler> there is some doc on how to do it
13:12:07 <mnasiadka> mmalchuk: it's in kolla release docs
13:12:27 <mnasiadka> So, continuing
13:12:36 <mmalchuk> if no hurry will do tomorow or at night
13:12:39 <mnasiadka> ansible-core 2.16 (and 2.17) require controller to be python3.10+
13:12:46 <mnasiadka> so we need to drop OpenEuler
13:13:01 <frickler> mmalchuk: https://docs.openstack.org/kolla/latest/contributor/release-management.html#maintained (needs an update in the active stable branches list)
13:13:01 <mnasiadka> we need to use python3.12 in Rocky 9 CI (only on the ansible controller)
13:13:02 <mmalchuk> finally)
13:13:16 <mnasiadka> so we'll also need to document this
13:13:23 <mmalchuk> frickler thnaks
13:13:30 <mmalchuk> *thanks
13:13:37 <mnasiadka> I'm working on getting the patch green - currently some hashicorp vault stuff is failing and zun
13:13:41 <frickler> but >= py3.10 should still be fine?
13:13:55 <mnasiadka> yes, Debian and Ubuntu is fine
13:14:05 <mnasiadka> and RHEL9 (so Rocky 9) has rpm packages for python3.12
13:14:15 <mnasiadka> so we're fine
13:14:26 <mnasiadka> and new Ubuntu will get something like 3.12 or later
13:14:43 <jovial> mnasiadka, Just thinking about the bump in kayobe. Will we have the usual issue with selinux/dnf bindings?
13:14:59 <mnasiadka> jovial: on the host ansible manages you can even have 3.6
13:15:07 <mnasiadka> so no issues with dnf bindings I guess
13:15:13 <mnasiadka> selinux I mean
13:15:20 <mnasiadka> hopefully
13:15:21 <jovial> But on the control host?
13:15:34 <mnasiadka> we'll see I guess
13:15:41 <jovial> I think we install a few packages
13:15:50 <mnasiadka> then we need to start working on it as well ;)
13:16:16 <jovial> Sounds like a plan
13:18:31 <jovial> I will knock up the kayobe change that depends on your patch :)
13:18:39 <mnasiadka> great, thanks
13:18:53 <mnasiadka> #topic Additional agenda (from whiteboard)
13:19:21 <mnasiadka> (mhiner)[July, 3rd]: Please review(Zuul is now happy): https://review.opendev.org/c/openstack/kolla-ansible/+/836941
13:21:39 <mnasiadka> mhiner: there are some unresolved comments - can you fix that?
13:21:59 <mnasiadka> I'll have a look in that patch later today
13:22:03 <mnasiadka> (SvenKieske)[June, 28th] I'd like some reviews and possibly RP+1 on these, please:
13:22:03 <mnasiadka> https://review.opendev.org/c/openstack/kolla-ansible/+/915403 (harden haproxy TLS config)
13:22:03 <mnasiadka> https://review.opendev.org/c/openstack/kolla-ansible/+/922840 (add a config validation check to haproxy)
13:22:03 <mnasiadka> https://review.opendev.org/c/openstack/kolla-ansible/+/921371 (sec hardening horizon by not mounting host tmp)
13:22:03 <mnasiadka> https://review.opendev.org/c/openstack/kolla-ansible/+/907977 (rmq precheck: also check fanout and reply queues)
13:24:48 <mhiner> mnasiadka: will add some replies, but in the end, Sven has to decide if he is happy with them.
13:25:08 <mnasiadka> mhiner: I mean it's easier for next reviewer if there are resolved comments :)
13:26:22 <mnasiadka> ok, went through Sven's patches, added -1 to each of them ;-)
13:26:34 <mnasiadka> (ihalomi)[3rd July] docker_worker refactor is now passing tests and is ready to review
13:26:34 <mnasiadka> https://review.opendev.org/c/openstack/kolla-ansible/+/908295
13:26:34 <mnasiadka> also refactor of container_facts is ready to be reviewed by others
13:26:34 <mnasiadka> https://review.opendev.org/c/openstack/kolla-ansible/+/911417
13:26:34 <mnasiadka> simple fix for bug found by kevko:
13:26:36 <mnasiadka> https://review.opendev.org/c/openstack/kolla-ansible/+/923231
13:26:43 <mnasiadka> ok then, let me have a look
13:27:34 <mnasiadka> first one is big, so RP+1 for later review
13:28:02 <mnasiadka> second one the same
13:28:47 <mnasiadka> and third needs a bug
13:29:07 <mnasiadka> jovial(3rd July)
13:29:07 <mnasiadka> Should we add a kayobe job to kolla CI?
13:29:07 <mnasiadka> Circular dependency so would need to be non-voting (I think)
13:29:07 <mnasiadka> Based on discussion in https://review.opendev.org/c/openstack/kolla-ansible/+/920368
13:29:10 <mnasiadka> ok then
13:29:45 <mnasiadka> jovial: how long is the job running? is it way longer than the regular aio kolla-ansible jobs?
13:30:37 <mnasiadka> just propose adding something, but I would prefer it's two jobs, not more :) so maybe aio rocky/ubuntu
13:30:37 <jovial> ~ 1h 08m 59s
13:30:48 <mnasiadka> and non-voting for starter
13:31:08 <frickler> are you talking about kolla really or k-a?
13:31:16 <mnasiadka> I think k-a
13:31:17 <ihalomi> mnasiadka: what do you mean it needs a bug?
13:31:21 <jovial> Yeah, I think it would be hard to make it voting as quite often we need a fix in kolla-ansible to get kayobe-ci to work
13:31:21 <mnasiadka> in kolla it doesn't make any sense
13:31:31 <mnasiadka> unless we have kayobe image build jobs, which we probably don't :)
13:31:55 <mnasiadka> jovial: well, it would make us merge things that don't break kayobe e.g. are backwards compatible :)
13:33:14 <jovial> True: might lead to use having to disable to get stuff into k-a occasionally (if its something that is external to k-a that has broken us)
13:33:41 <mnasiadka> ok, so adding two kayobe aio jobs into kolla-ansible - that makes sense
13:33:56 <mnasiadka> regarding frickler's earlier question :)
13:34:07 <jovial> I just wanted to check that it wouldn't get immediately rejected if I proposed something :)
13:34:41 <mnasiadka> we don't immediately reject anything, you can get grilled and asked what are you doing - but not an immediate -2 ;-)
13:35:23 <jovial> For kolla: We do now build the container images: https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_abc/922795/5/check/kayobe-seed-images-rocky9/abcc463/primary/ansible/overcloud-container-image-build
13:35:55 <mnasiadka> Maybe that's a second step, but don't know how would we be able to break kayobe image build :)
13:36:05 <jovial> Just what I was thinking :D
13:36:48 <mnasiadka> ok then, that's it from the whiteboard
13:37:01 <mnasiadka> #topic Open discussion
13:37:04 <mnasiadka> Anybody, anything?
13:37:09 <r-krcek> Hi!   I am thinking of moving kolla-asible CLI into python. m.nasiadka recommended to use cliff package - like is used in kayobe-cli or openstack-cli. After reading the docs I noticed that the command structure is different from kolla-ansible. (the commands have subcommands eg. kayobe baremetal compute manage)  Would you be more inclined if the new CLI would be are similar to the old one or if it followed the convention used in kayobe-cli and op
13:37:09 <jovial> OK, so I'll try and submit a patch to add the AIO job when I get a bit of time :)
13:38:42 <jovial> How many people use kolla-ansible CLI?
13:38:43 <frickler> r-krcek: seems your message was cut off after "kayobe-cli and op"
13:39:08 <r-krcek> continuation: For example "kolla-ansible bifrost deploy" instead of "kolla-asnible deploy-bifrost" or "kolla-ansible mariadb backup" and "kolla-ansible mariadb recovery" "kolla-ansible containers deploy" and "kolla-ansible containers stop" etc.  Plus some features like multiple environments and hooks are marked as experimental. Do you think it is worth it/wanted to also add these features? What do you think?
13:39:42 <jovial> Oh my bad, was thinking of https://opendev.org/openstack/kolla-cli
13:39:46 <mnederlof> doesn´t everyone use the kolla-ansible cli?
13:40:07 <mnederlof> or is it recommended to implement the roles in your own playbooks?
13:40:48 <mnasiadka> Well, from code perspective - I would prefer we have similar code for CLI - so it's easier to maintain as a project
13:40:55 <jovial> mnederlof, Think this was confusion on my part, this is about writing the kolla-ansible bash script in python I guess
13:41:07 <mnasiadka> yes, that's about rewriting bash script to python
13:41:14 <mnasiadka> and if we have something working for Kayobe
13:41:19 <mnasiadka> then for me it's a no brainer task
13:41:23 <mnasiadka> or is it just me? :)
13:42:23 <mnederlof> i am fine with migrating it to python, what comes to mind is that it might be harder to have live output while ansible runs
13:42:43 <mnasiadka> well, we have that working in Kayobe
13:42:47 <mnederlof> awesome
13:42:53 <mnasiadka> so let's not reinvent the whell
13:42:55 <mnasiadka> wheel
13:42:58 <mnasiadka> r-krcek: what do you say?
13:43:16 <mnasiadka> r-krcek: I wouldn't add any features now - just achieve 1:1 feature parity
13:43:33 <jovial> +1
13:43:34 <mnederlof> as a first release anyway :)
13:43:37 <mnasiadka> but in the past kolla-cli also supported multiple environments (but not in the same way as Kayobe does that today)
13:43:44 <r-krcek> I agree qith m.nasiadka. Lets not reinvent the wheel.  :)
13:44:14 <mnederlof> i also have another question, regarding the use of systemd for starting/stopping the docker containers, does anyone know why this was changes?
13:44:16 <mnederlof> *changed
13:44:24 <mnasiadka> because we also support podman
13:44:29 <mnasiadka> and that was the only sensible way
13:44:35 <mnederlof> ah, ok
13:44:41 <r-krcek> Okay, so for now just as close as I can get it to 1:1 with the old interface. Thanks for the input
13:45:55 <mnederlof> we seem to have some issues with systemd, when a container is stopped continuesly (for example a compute node that is restarted, and does not have its compute_id file yet and is still known in the nova database)
13:46:11 <mnasiadka> I think we have a bug around compute_id and managing that properly
13:46:18 <mnasiadka> currently we leave that to Nova
13:46:28 <mnasiadka> and compute_id file gets put in /var/lib/nova
13:46:38 <mnederlof> yes, in the docker volume
13:46:52 <mnasiadka> well, docker volume should be there without any problems
13:46:56 <mnederlof> yes
13:47:08 <mnasiadka> I was thinking that some people put that on NFS - and if that's not mounted or unavailable you can get into issues
13:47:50 <jovial> I guess you could use add a dependency to docker to not start until the storage is mounted for that case?
13:47:52 <mnasiadka> Have you raised a bug and attached some logs?
13:48:09 <mnederlof> the compute_id file is not really the problem though, a restarting container i can fix, but having to reboot the server because systemd is on the fence is kind of annoying (also because the whole system becomes sluggish)
13:48:28 <mnederlof> no not yet, as i am not sure what the trigger is yet
13:48:31 <mnederlof> will do that though
13:48:35 <mnasiadka> well, if we can find the cause for that behaviour - maybe we can fix that
13:48:46 <mnederlof> but was thinking about making systemd optional
13:48:46 <mnasiadka> feel free to raise a bug and leave the link here - so we can have a look
13:49:03 <mnederlof> when using docker
13:49:36 <mnasiadka> well, that's sort of a step back - we haven't observed any issues with systemd approach yet
13:49:48 <mnasiadka> so raise that bug and we can continue in there :)
13:49:53 <mnederlof> yes, will do that :)
13:50:01 <mnasiadka> thanks
13:50:07 <mnasiadka> ok, I think that's it for today
13:50:13 <mnasiadka> Thank you all for coming - see you next week!
13:50:15 <mnasiadka> #endmeeting