15:00:24 <noonedeadpunk> #startmeeting openstack_ansible_meeting
15:00:24 <opendevmeet> Meeting started Tue Sep  3 15:00:24 2024 UTC and is due to finish in 60 minutes.  The chair is noonedeadpunk. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:00:24 <opendevmeet> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
15:00:24 <opendevmeet> The meeting name has been set to 'openstack_ansible_meeting'
15:00:30 <noonedeadpunk> #topic rollcall
15:00:32 <noonedeadpunk> o/
15:00:35 <jrosser> o/ hello
15:01:56 <noonedeadpunk> #topic office hours
15:02:05 <noonedeadpunk> so we have couple of things for dicsussion
15:02:19 <opendevreview> Jonathan Rosser proposed openstack/openstack-ansible-plugins master: Add setup_hosts playbook to plugins collection.  https://review.opendev.org/c/openstack/openstack-ansible-plugins/+/927826
15:02:23 <noonedeadpunk> Noble support is almost here from what I see
15:02:28 <noonedeadpunk> #link https://review.opendev.org/c/openstack/openstack-ansible/+/924342
15:02:43 <jrosser> sort of - i would say yes as far as the integrated repo is concerned
15:02:55 <noonedeadpunk> but job is failing multiple times in a row now, but every time in a different way
15:03:01 <jrosser> probably no as far as all additional services are concerned
15:03:12 <noonedeadpunk> yeah, that's true as well
15:03:21 <jrosser> we should do some work on CI stability
15:03:29 <jrosser> i have been trying to keep notes on common failures
15:03:32 <NeilHanlon> hiya
15:03:45 <jrosser> like failing to get u-c, image download errors etc
15:03:58 <noonedeadpunk> I've spotted bunch of mirrors issues with RDO lately as well
15:04:07 * NeilHanlon hopes for few rocky issues
15:04:13 <jrosser> but there is also a rumble of tempest failures, perhaps with more often than not it being keystone
15:04:15 <noonedeadpunk> were some :D
15:04:23 <jrosser> andrewbonney: ^ you were looking at failures too a bit I think?
15:04:32 * NeilHanlon plugs his ears and pretends he didn't hear anything
15:04:53 <jrosser> and the mcapi job is extremely troublesome, which needs more investigation
15:04:56 <noonedeadpunk> NeilHanlon: actually we've also discussed with infra folks Rocky mirrors
15:05:08 <jrosser> but on the surface that looks like nothing at all to do with magnum causing the errors
15:05:14 <NeilHanlon> yeah i remember some message from last month or so... travelling took a lot out of me
15:05:20 <noonedeadpunk> seems they do have space on afs share now and were fine adding them
15:05:23 <NeilHanlon> i will try and restart that convo
15:05:51 <noonedeadpunk> yeah, would make sense, as CentOS testing was pulled of as a whole due to experiencing quite some issues
15:05:59 <noonedeadpunk> and rocky was discussed as a replacement
15:06:15 <NeilHanlon> right
15:06:32 <noonedeadpunk> about capi jobs - I frankly did not look into these at all
15:06:38 <noonedeadpunk> as still barely get the topic
15:06:50 <noonedeadpunk> though coming closer and closer by internal backlog to it
15:07:30 <noonedeadpunk> Another thing that you raised my attention to - is changing a way of how uwsgi is supposed to be served
15:07:45 <noonedeadpunk> and puling off wsgi scripts from service setup scripts
15:07:59 <noonedeadpunk> So this bump will totally fail on these changes
15:08:03 <noonedeadpunk> #link https://review.opendev.org/c/openstack/openstack-ansible/+/927841
15:08:45 <jrosser> hopefully we can make some depends-on patches and work through what is broken fairly easily
15:09:04 <noonedeadpunk> yeah
15:09:11 <noonedeadpunk> and with that test noble I hope
15:09:25 <opendevreview> Merged openstack/openstack-ansible master: Verify OS for containers installation  https://review.opendev.org/c/openstack/openstack-ansible/+/925974
15:09:55 <noonedeadpunk> we also need to come up with release highlights
15:10:00 <jrosser> do we have anything big left to fix/merge this cycle?
15:10:18 <NeilHanlon> i guess i will also probably start on rocky 10 experimental jobs at some point. i need to check up with RDO folks first
15:10:26 <jrosser> deb822 is one thing, but i think thats now understood and is just a question of doing the other places
15:10:50 <noonedeadpunk> looking through our ptg doc
15:10:52 <noonedeadpunk> #link https://etherpad.opendev.org/p/osa-dalmatian-ptg
15:11:05 <NeilHanlon> goodness, it's almost PTG again isnt it..
15:11:07 <noonedeadpunk> and realizing I failed to work on most interesting topic for myself so far
15:11:13 <jrosser> but it would be quite good to be able to spend the rest of the cycle getting existing stuff merged and doing tidy/up & CI fixing
15:11:14 <noonedeadpunk> NeilHanlon: it really is....
15:11:26 <jrosser> we have had a couple of times now with a real big rush for release
15:11:30 <noonedeadpunk> jrosser: yes, exactly. I don't aim to bring anything new
15:11:41 <noonedeadpunk> really want to have a coordinated release as a feature freeze
15:12:13 <jrosser> i would say we are basically there apart from finishing a few things
15:12:20 <noonedeadpunk> so about topics: deb822, noble, playbooks into collection
15:12:36 <jrosser> yeah
15:12:50 <jrosser> i will try to find time soon to revisit the deb822 stuff
15:13:02 <NeilHanlon> oh i forgot if i mentioned it but i do have a working incus for rocky 9
15:13:13 <NeilHanlon> https://copr.fedorainfracloud.org/coprs/neil/incus/
15:13:14 <noonedeadpunk> oh, that's really nice.
15:13:22 <jrosser> noble is potentially a big job, as also i think we have some still broken roles
15:13:26 <noonedeadpunk> we should try to look into that for 2025.1 I guess
15:13:32 <NeilHanlon> agreed
15:13:59 * NeilHanlon reads up on what deb822 is
15:14:06 <jrosser> and for playbooks->collection - we should decide how far we go this cycle
15:14:08 <noonedeadpunk> yeah, these are broken ones
15:14:10 <noonedeadpunk> #link https://review.opendev.org/q/topic:%22osa/frist_host_refactoring%22+status:open
15:14:24 <noonedeadpunk> jrosser: I'd go all-in
15:14:27 <jrosser> like is -hosts and -infra enough and we treat -openstack as further work?
15:14:56 <noonedeadpunk> I can get some time to finalize jsut in case
15:15:04 <jrosser> ok - i have kind of lost where we got up to as it has taken so very long to merge the initial stuff
15:15:41 <jrosser> there will be some remaining common-tasks / common-playbooks i expect
15:15:50 <noonedeadpunk> yeah, it took quite long for reviews as well to ensure that all changes to playboks were moved as well
15:17:32 <noonedeadpunk> so far good question is what to do with things like ceph playbooks
15:19:21 <noonedeadpunk> but it looks like that most of things you moved already anyway:)
15:19:26 <noonedeadpunk> so it's good
15:19:51 <noonedeadpunk> And there's also - what to do with things like that: https://opendev.org/openstack/openstack-ansible/src/branch/master/playbooks/listening-port-report.yml
15:20:08 <noonedeadpunk> I assume you're using this?
15:20:42 <jrosser> that was very useful in the time of working on bind-to-mgmt
15:20:55 <jrosser> but i think actually there is an ansible module to do the same now
15:21:26 <jrosser> https://docs.ansible.com/ansible/2.9/modules/listen_ports_facts_module.html
15:21:45 <noonedeadpunk> yeah
15:22:27 <noonedeadpunk> ok, so overall the list sounds doable - noble, wsgi_scripts and playbooks
15:22:39 <jrosser> i think so
15:23:24 <jrosser> the magnum stuff is ok - but we do risk making a release that includes installing stuff from github.com/jrosser fork which i don't like
15:23:27 <jrosser> mnaser: ^
15:25:18 <noonedeadpunk> Btw there was 1 bug report I wanted to check on, but failed so far
15:25:20 <noonedeadpunk> #link https://bugs.launchpad.net/openstack-ansible/+bug/2078552
15:26:12 <noonedeadpunk> I believe there's a race condition in there, as in case `rabbitmqctl cluster_status` exits with error code, which triggeres assert failure, then we probably should not attempt to run it to get flags either
15:26:39 <noonedeadpunk> But I didn't look in the code, but I guess expectation for recovery in case of cluster failure is fair
15:27:12 <noonedeadpunk> I was thinking though if it would make sense to add another flag like `ignore_cluster_state` as we have in mariadb
15:27:25 <jrosser> andrewbonney: you may have thoughts on this ^
15:27:55 <noonedeadpunk> but then it might go too far, and raise a question if mnesia should be preserved with that flag or not
15:27:57 <jrosser> time going backwards is really bad though :)
15:28:00 <andrewbonney> Yeah, I'll try and look tomorrow, context switch is too hard right now
15:28:13 <noonedeadpunk> oh yes, it's not good :D
15:28:38 <noonedeadpunk> I can get how that happened though...
15:28:42 <noonedeadpunk> or well
15:29:05 <noonedeadpunk> I spotted couple of times, that after reboot chrony somehow does not startup properly from time to time
15:29:09 <jrosser> openstack doesnt support 24.04 for D does it?
15:29:18 <noonedeadpunk> no, they're trying master
15:29:39 <noonedeadpunk> there was another report: https://bugs.launchpad.net/openstack-ansible/+bug/2078521
15:29:41 <jrosser> right - so i still think we need to be careful what message we give out
15:30:21 <noonedeadpunk> yeah, I explained support matrix in the previous one
15:30:36 <noonedeadpunk> so folk is trying to beta test on master and report back findings
15:30:56 <noonedeadpunk> just pretty much missed collection dependency I guess
15:32:03 <jrosser> indeed - the noble topic is really only just all merged now
15:32:34 <noonedeadpunk> but dunno... anyway, overall issue description looks reasonable enough t o double check
15:34:09 <noonedeadpunk> there was another one, but I feel like it's a zun issue
15:34:12 <noonedeadpunk> #link https://bugs.launchpad.net/openstack-ansible/+bug/2078482
15:34:34 <noonedeadpunk> so at worst we can mark it as invalid for osa
15:36:47 <jrosser> interesting venv paths in that bug report
15:37:39 <noonedeadpunk> indeed....
15:37:59 <noonedeadpunk> ah
15:38:08 <noonedeadpunk> I guess it's just top of the 2024.1
15:38:25 <noonedeadpunk> and pbr detects version tag as `stable/2024.1`
15:38:31 <noonedeadpunk> though I would not expect that happening
15:38:48 <jrosser> i thought you still got the previous tag with -dev<big-number> in that case
15:39:04 <noonedeadpunk> it used to be that way for sure, yes
15:39:06 <jrosser> well, some number
15:40:10 <noonedeadpunk> but technically one can override version as well
15:40:45 <noonedeadpunk> but that's pretty much it then
15:43:55 <noonedeadpunk> ah, we have another "bug" on master (and 2024.1 I guess)
15:44:05 <noonedeadpunk> we have conflicting MPMs for Apache between services
15:44:43 <noonedeadpunk> like repo and keystone asking for event and horizon and skyline for event
15:44:46 <noonedeadpunk> or smth like that
15:44:51 <jrosser> actually this is something we should fix
15:44:56 <noonedeadpunk> so re-running playbooks result in fialures
15:45:10 <noonedeadpunk> things went completely off with repo actually
15:45:23 <noonedeadpunk> yeah, I was just thinking about best way for that
15:45:26 <jrosser> thats only in master though currently?
15:45:58 <noonedeadpunk> well, in stable you can shoot into your leg as well
15:46:11 <noonedeadpunk> like - override https://opendev.org/openstack/openstack-ansible-os_keystone/src/branch/master/defaults/main.yml#L235
15:46:56 <noonedeadpunk> but then - https://opendev.org/openstack/openstack-ansible-os_skyline/src/branch/master/vars/debian.yml#L31-L34
15:47:35 <noonedeadpunk> and https://opendev.org/openstack/openstack-ansible-os_horizon/src/branch/master/vars/debian.yml#L61-L64
15:48:13 <noonedeadpunk> so this all leans towards apache role eventually
15:48:50 <jrosser> yes agreed
15:49:22 <noonedeadpunk> but also I think this should be still be backportable at first...
15:51:09 <noonedeadpunk> ah, and also what I found yesterday - is a bug in neutron handlers for l3 - these 2 things just doens not work on modern kernels https://opendev.org/openstack/openstack-ansible-os_neutron/src/branch/master/handlers/main.yml#L33-L75
15:51:40 <noonedeadpunk> but also I'm not sure what's meant under `pgrep neutron-ns-meta`
15:54:04 <noonedeadpunk> I'm not sure though if worth including apache thing in this release.. I guess not, but for 2025.1
16:00:00 <noonedeadpunk> #endmeeting