14:00:08 <mnasiadka> #startmeeting kolla
14:00:08 <opendevmeet> Meeting started Wed Feb 21 14:00:08 2024 UTC and is due to finish in 60 minutes.  The chair is mnasiadka. Information about MeetBot at http://wiki.debian.org/MeetBot.
14:00:08 <opendevmeet> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
14:00:08 <opendevmeet> The meeting name has been set to 'kolla'
14:00:10 <mnasiadka> #topic rollcall
14:00:11 <mnasiadka> o/
14:00:22 <mmalchuk> \o
14:00:26 <jovial> o/
14:00:37 <mattcrees> o/
14:01:00 <frickler> \o
14:01:08 <kevko> \o/
14:01:32 <SvenKieske> o/
14:01:39 <SvenKieske> (time flies)
14:01:40 <halomiva> \o
14:02:01 <opendevreview> Michal Nasiadka proposed openstack/kolla master: opensearch: set OPENSEARCH_INITIAL_ADMIN_PASSWORD  https://review.opendev.org/c/openstack/kolla/+/909644
14:02:48 <mnasiadka> #topic agenda
14:02:48 <mnasiadka> * CI status
14:02:49 <mnasiadka> * Release tasks
14:02:49 <mnasiadka> * Regular stable releases (first meeting in a month)
14:02:49 <mnasiadka> * Current cycle planning
14:02:50 <mnasiadka> * Additional agenda (from whiteboard)
14:02:50 <mnasiadka> * Open discussion
14:02:52 <mnasiadka> #topic CI status
14:03:02 <mnasiadka> Soo... OpenSearch 2.12 has broken our builds
14:03:15 <mnasiadka> fixing with 909644 - once it passes, I'll add some more meat in commit msg
14:03:22 <mnasiadka> and raise a proper bug for backporting
14:03:29 <mnasiadka> any other CI breakages?
14:03:35 <mnasiadka> jovial: you mentioned something in Kayobe?
14:04:31 <jovial> Yeah, seed jobs are failing
14:04:35 <mnasiadka> ok
14:04:48 <mnasiadka> and Cinder seems to be finally fixed (upgrade cephadm jobs break)
14:04:56 <jovial> I've marked it as red in the whiteboard
14:05:00 <mnasiadka> so - new breakages, old breakages, fun
14:05:04 <mnasiadka> #topic Release tasks
14:05:27 <bbezak> o/
14:05:33 <mnasiadka> This week is R-6 - Final release for non-client libraries
14:05:54 <mnasiadka> next week is feature freeze (but not for Kolla)
14:06:10 <mnasiadka> #topic Current cycle planning
14:06:20 <mnasiadka> Anybody wants to discuss any patches?
14:06:27 <kevko> yep
14:06:46 <kevko> just trivial one https://review.opendev.org/c/openstack/kolla-ansible/+/908429
14:07:17 <jovial> Nothing to discuss yet, but I was going to push a WIP version of podman support for kayobe for some initial comments
14:07:24 <mmalchuk> https://review.opendev.org/c/openstack/kolla-ansible/+/907495 can be merged, fixed depends-on
14:08:03 <mmalchuk> please review trivial: https://review.opendev.org/c/openstack/kayobe/+/909205
14:08:21 <mmalchuk> and https://review.opendev.org/c/openstack/kayobe/+/909113
14:09:04 <mmalchuk> and last trivial: https://review.opendev.org/c/openstack/kolla-ansible/+/907254
14:09:43 <mnasiadka> discuss, not trivial patches concert, but whatever :)
14:09:57 <jovial> Was also going to try bumping kayobe ansible to 2.16 to follow: https://review.opendev.org/c/openstack/kolla-ansible/+/907522. I think I managed to get the issue we hit in wcmatch (the globbing library) fixed upstream. We hit this when using newer ansible: https://github.com/facelessuser/wcmatch/issues/210
14:10:12 <frickler> mnasiadka: +1 was about to ask the same
14:11:13 <SvenKieske> just my whiteboard topic, but that's for later I guess :)
14:11:13 <mnasiadka> jovial: seems a patch was released 20 hours ago?
14:12:01 <jovial> Indeed, so hopefully just a case of bumping the dependencies. I've tested newer 2.15.x, but haven't tried 2.16 yet.
14:12:30 <mnasiadka> ok
14:13:04 <mnasiadka> OpenSearch breakage broke the opensearch CI jobs in kolla-ansible (because for some bloody reason we have opensearch-dashboards 2.12 and opensearch 2.11)
14:13:15 <mnasiadka> but it will get fixed when we fix the build
14:13:36 <mnasiadka> not asking why the heck anybody would want to do a post script in rpm/deb based on env var (for setting initial password)
14:13:43 <mnasiadka> but that's beyond my comprehension
14:14:16 <mnasiadka> #topic Additional agenda (from whiteboard)
14:14:18 <mnasiadka> Let's go forward
14:14:20 <SvenKieske> mhm, very unfortunate that we have to set that password, I take it there's no way around that?
14:14:31 <SvenKieske> but we can discuss this also later/in the patch
14:14:32 <mnasiadka> SvenKieske: no
14:14:35 <mnasiadka> (SvenKieske): Do we want to officially support ansibles --limit option for upgrades? it's at least broken for some services e.g. https://bugs.launchpad.net/kolla-ansible/+bug/2054348
14:14:58 <mnasiadka> So, from my perspective - supporting --limit is complicated - as in - in most cases it will work
14:15:02 <mnasiadka> unless it doesn't
14:15:07 <mnasiadka> and we can't really test that in CI
14:15:44 <SvenKieske> yeah, so I'm open to just document that it doesn't work for most cases, but imho we should at least do that, because I ran into issues and it's always good if I can point to docs saying: that's not supported :D
14:15:54 <opendevreview> Maksim Malchuk proposed openstack/kolla-ansible master: Fix 'cinder-backup' service when Swift with TLS enabled  https://review.opendev.org/c/openstack/kolla-ansible/+/907495
14:16:13 <jovial> Were you limiting to a single controller?
14:16:27 <SvenKieske> I can volunteer to do a docs patch for that. if there is consensus around this? or do we want to support it for some use cases at least (e.g. nova?)
14:17:05 <SvenKieske> limiting to a single compute node, but nova-api upgrade stuff breaks then, see the bug report, I hope everything is included there (it was late)
14:17:38 <kevko> hmm, that bug in launchpad regarding limit ...it's also because in k-a we are using host[0] and register
14:18:14 <frickler> IMO limiting to computes needs to be supported, so we should look into fixing that, even if we don't have a CI job for that
14:18:14 <SvenKieske> the thing is, the nova-upgrade task registers results as a host var which aren't accessible by other hosts, as a result the playbook stops with an error when that host is not part of the actual play.
14:18:19 <jovial> SvenKieske, I don't think the bug mentions the limit you used in the bug report, might be handy to add
14:18:24 <mattcrees> I'd say there's no harm in fixing bugs like the one Sven found, but given we have no way to regularly test it we shouldn't officially be supporting it.
14:18:26 <kevko> frickler: agree
14:19:38 <mnasiadka> Well, I agree that 1. we should state in the docs that running with limit is not tested 2. make that work 3. accept bugs for --limit with low priority (or wishlist priority) stating it's not tested in CI and requires more work
14:19:47 <SvenKieske> okay, makes sense, because I agree limiting to certain compute nodes is a often used feature
14:19:52 <mnasiadka> Does that make sense?
14:20:00 <frickler> mnasiadka: +1
14:20:10 <SvenKieske> so I guess we should refactor "register:" stuff to "set_fact" instead
14:20:11 <mattcrees> +1 from me
14:20:18 <SvenKieske> mnasiadka: +1 from me as well
14:20:22 <mnasiadka> Ok, who's the volunteer to do the docs update?
14:20:45 <SvenKieske> I can take care of 1.) push a patch that states that --limit is currently not tested
14:21:09 <SvenKieske> do we document that just in the upgrade docs or anywhere else?
14:21:19 <mnasiadka> Support matrix maybe?
14:22:22 <mnasiadka> Anyway, try to find a good spot and let's discuss in Gerrit
14:22:43 <mnasiadka> next one
14:22:45 <mnasiadka> (halomiva): bump version of docker-py so its supports cgroupns for this change https://review.opendev.org/c/openstack/kolla-ansible/+/908295
14:23:17 <SvenKieske> okay
14:23:36 <mnasiadka> halomiva: bump to 7.0.0?
14:23:42 <halomiva> is it okay bumping it from 3.0 to 6.0 ?
14:23:56 <mnasiadka> to 6.0 yes, but lower than 7.0
14:24:12 <mnasiadka> there was a bug with the url syntax checker there that failed our Kolla builds
14:24:21 <mnasiadka> so I anticipate similar in k-a
14:24:26 <mnasiadka> and they haven't still released 7.0.1
14:24:31 <halomiva> okay, that should be enough, is there also some minimal version of docker that we support?
14:24:59 <halomiva> because there might be problem that docker api doesnt support it either before 20.1 i think
14:24:59 <opendevreview> Jake Hutchinson proposed openstack/kayobe master: Register baremetal compute nodes in Ironic.  https://review.opendev.org/c/openstack/kayobe/+/909671
14:25:20 <mnasiadka> halomiva: https://github.com/openstack/kolla-ansible/blob/d30fb56c2aabc04fdc922a16cb85194c7e587459/ansible/roles/prechecks/vars/main.yml#L2
14:25:22 <mnasiadka> we have this
14:25:27 <mnasiadka> so it might need a bump
14:26:06 <halomiva> okay i will test it with 6.0 and 20.10 once i finish unit tests
14:26:20 <mnasiadka> another thing is refactor should not require a version bump in theory...
14:26:26 <jovial> Incidentally, docker-py was also breaking the kayobe image build job. I wonder if I should fix to <7 as the new release doesn't seem forthcoming.
14:26:59 <mnasiadka> jovial: it breaks when you push to a registry with non-standard port - so yes, pinning is advised ;-)
14:27:18 <mnasiadka> halomiva: is there any way we could do the refactor without bumping versions, and then just bump versions for some additional functionality?
14:27:20 <halomiva> in theory yes but the reason low level API is used is because there wasnt higher level capable of doing it at that time
14:27:29 <SvenKieske> mhm, should we maybe get the precheck min versions from our requirements.txt? why have the versions in multiple files? it's harder to maintain, no? (it might be not trivial to parse the min version though)
14:28:13 <SvenKieske> that should be the issue about the broken docker tag parsing with regards to registry ports: https://github.com/docker/docker-py/issues/3195
14:28:36 <jovial> mnasiadka: K, sounds like a good plan as it could be breaking people already and it would be good to get that job back to voting
14:28:42 <SvenKieske> fix is here: https://github.com/docker/docker-py/commit/3ec5a6849a6cad4c5f5f3bafb5f74b5827fec14c
14:28:58 <mnasiadka> ok, let's go to the next one
14:29:08 <mnasiadka> (halomiva): refactor of kolla_container_facts and volume_facts metioned here https://review.opendev.org/c/openstack/kolla-ansible/+/905837
14:29:16 <halomiva> mnasiadka: like we can do it without supporting cgroupns but then i wasnt able to run virtuals because of libvirt issue
14:29:46 <halomiva> basically as I said in the last comment, what do you think about it?
14:30:26 <mnasiadka> halomiva: I like the idea in the last comment - let me reply there
14:31:09 <mnasiadka> ok then, that was the last topic
14:31:13 <mnasiadka> #topic Open discussion
14:31:17 <mnasiadka> Anybody anything?
14:33:37 <SvenKieske> doesn't seem to be the case :)
14:34:07 <frickler> oh, just one thing to mention
14:34:23 <frickler> PTL and TC nomination period has started
14:34:37 <mnasiadka> Yes, I signed up for Kolla
14:34:39 <mnasiadka> Again
14:34:50 <SvenKieske> that sounds a little tired? :)
14:35:31 <mnasiadka> Naah
14:35:51 <mnasiadka> I like doing that, from other perspective maybe some fresh blood would be good
14:36:01 <mnasiadka> We can discuss on the PTG if anybody wants to run for E
14:37:28 <SvenKieske> we could try the distributed PTL stuff, I don't feel I understand all the organizational stuff enough just yet, I know it's mostly documented :)
14:38:00 <mmalchuk> I want, but can't ;)
14:38:21 <mnasiadka> distributed PTL looks like a way to tell nobody is responsible ;)
14:39:21 <mnasiadka> Ok, I guess that's enough for today
14:39:26 <mnasiadka> Thank you for coming and see you next week!
14:39:28 <mnasiadka> #endmeeting