15:00:42 <noonedeadpunk> #startmeeting openstack_ansible_meeting
15:00:42 <opendevmeet> Meeting started Tue May 20 15:00:42 2025 UTC and is due to finish in 60 minutes.  The chair is noonedeadpunk. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:00:42 <opendevmeet> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
15:00:42 <opendevmeet> The meeting name has been set to 'openstack_ansible_meeting'
15:00:47 <noonedeadpunk> #topic rollcall
15:00:48 <noonedeadpunk> o/
15:01:28 <jrosser> o/ hello
15:02:58 <noonedeadpunk> #topic office hours
15:03:06 <noonedeadpunk> so I have good and bad news kinda
15:03:20 <noonedeadpunk> good one, is that we've branched 2025.1
15:03:41 <noonedeadpunk> And we're pretty much in time for the release in 2 weeks
15:04:26 <noonedeadpunk> despite most things were not done at last minute, we still having a release with no time to spare
15:04:45 <noonedeadpunk> And I wanted to say with way lower amount of stress as well
15:04:52 <noonedeadpunk> if not... EL10 situation
15:05:06 <jrosser> i think we just have to let that go for a while
15:05:18 <jrosser> seems like something that will eventually come (CI nodes)
15:05:35 <noonedeadpunk> and here bad news are coming. As python 3.11 is the minimal requirement, I did include drop of all EL9 related jobs to
15:05:39 <noonedeadpunk> #link https://review.opendev.org/c/openstack/openstack-ansible/+/950261
15:05:56 <jrosser> i did a patch also to drop jammy
15:06:11 <noonedeadpunk> I actually decided it to squash with this one
15:06:17 <noonedeadpunk> as it would be conflicting anyway
15:06:20 <jrosser> no problem, yes
15:06:38 <noonedeadpunk> I did not do any deep clean-up for EL though
15:06:48 <noonedeadpunk> IT was more of docs/matrix/jobs thing
15:07:02 <noonedeadpunk> and I wanna start looking locally at el10 support asap
15:07:30 <opendevreview> Merged openstack/openstack-ansible stable/2025.1: Update .gitreview for stable/2025.1  https://review.opendev.org/c/openstack/openstack-ansible/+/950417
15:07:31 <noonedeadpunk> nasty part, that we will have to backport it for upgrade reasons...
15:08:13 <noonedeadpunk> and I really don't like this specific part of it
15:08:42 <noonedeadpunk> But, well... we have what we have I guess...
15:08:53 <jrosser> we certainly do
15:08:59 <noonedeadpunk> also I left rocky 9 molecule jobs
15:09:25 <noonedeadpunk> as where they are used, they should not hurt just for sanity checks of the codepath
15:10:02 <jrosser> i am not sure how much i like having to backport a ton of stuff either
15:10:32 <jrosser> it is currently so much work to keep up with the EOL / unmaintained fall out that i just don't know if it is a good idea
15:10:33 <noonedeadpunk> it can be not _that_ bad if it's the next thing we land
15:10:58 <noonedeadpunk> as more or less clean backports can be done with a single button in gerrit
15:11:27 <noonedeadpunk> but then we're dropping an upgrade path, which is also not good
15:12:04 <noonedeadpunk> and I think it's me to blame, as I didn't manage to work on support in time, totally missed that we can't do this next cycle
15:13:00 <opendevreview> Dmitriy Rabotyagov proposed openstack/openstack-ansible master: Start development of 2025.2 (Flamingo)  https://review.opendev.org/c/openstack/openstack-ansible/+/950261
15:13:06 <jrosser> thats not entirely fair - to land support for a whole OS with major changes that maybe you don't even use for deployments
15:13:25 <jrosser> imho we really need more hands on the rh derived stuff in total
15:14:06 <jrosser> the debian and ubuntu OS upgrades are almost a no-op in comparison
15:14:08 <opendevreview> Dmitriy Rabotyagov proposed openstack/openstack-ansible master: Start development of 2025.2 (Flamingo)  https://review.opendev.org/c/openstack/openstack-ansible/+/950261
15:14:30 <noonedeadpunk> well, I don't use, but it come really handy for me couple of times in arguments
15:15:24 <noonedeadpunk> where it was - "we want EL, as we trust in it" - "well, maybe let;s start with deb, and if it won't work - we can always seamingly migrate to EL?" - "lgtm" kind of conversatrions
15:16:20 <noonedeadpunk> we do OS upgrades through re-install anyway btw, so if not mentioning amount of changes required in code/packages/etc - for operators upgradfe process is the same
15:16:26 <noonedeadpunk> but I totally get what you're saying
15:16:32 <noonedeadpunk> and I agree with it
15:16:54 <noonedeadpunk> just from the "product" prespective better to have such upgrade release...
15:17:00 <NeilHanlon> ah!
15:17:03 <jrosser> yes thats very true
15:17:05 * NeilHanlon got distracted by prometheus
15:17:07 <NeilHanlon> sorry :D
15:17:13 <jrosser> easy to happen :)
15:18:07 <noonedeadpunk> we're discussing dropping Rocky support :D
15:18:10 <NeilHanlon> sry reading back :D
15:18:18 <noonedeadpunk> so you haven't missed anything important :p
15:18:21 <NeilHanlon> hah
15:18:35 <NeilHanlon> no it's OK, i mean.. I get it
15:18:56 <jrosser> seems maybe the immediate issue is getting more horsepower on the CI node issue
15:19:06 <noonedeadpunk> But in seriosness, I wanna start doing patches this week
15:19:12 <jrosser> as anyone can do local dev with a cloud image and make patches
15:19:22 <jrosser> but we can't test/merge anything right now sensibly
15:19:40 <NeilHanlon> and I agree w/ jrosser about it not being really all on your plate noonedeadpunk especially when it's _not_ something you deploy actively
15:19:59 <NeilHanlon> but passing blame around isn't probably productive :D
15:20:39 <NeilHanlon> the immediate need is to fix for our current users and make sure we can keep pushing code we are confident in because it's passing CI
15:20:43 <noonedeadpunk> I'd actually vote on landing EL10 even without testing.... Just openly stating that everywhere to manage expectations
15:21:23 <noonedeadpunk> as I believe that CI might be solved sooner then later
15:21:35 <NeilHanlon> another possible interesting thing re: major version upgrades is Oracle recently did this: https://github.com/openela/leapp-repository
15:21:43 <noonedeadpunk> we can add jobs as NV to find issues and them set as voting
15:22:10 <NeilHanlon> but that's tangential at this point as it's just more work we'd need to do to maintain actors for doing openstack upgrades atop rocky and i prefer not doing that anyways
15:22:33 <noonedeadpunk> right, I prefer not upgrading ubuntu either, whenever I can
15:22:53 <noonedeadpunk> jsut re-install is way more trivial if you have all internal things automated anyway
15:23:05 <noonedeadpunk> (and take less time)
15:23:11 <jrosser> i think i miswrote earlier about OS upgrades, i really meant introducing jobs for a new ubuntu/debian tends to be very trivial
15:23:20 <NeilHanlon> right, that's sorta the whole reason we use ansible !
15:23:37 <noonedeadpunk> jrosser: yeash, that's true
15:23:38 <NeilHanlon> jrosser: yeah I think that also threw me off but i still wanted to mention it's technically more possible now
15:24:02 <noonedeadpunk> apparmor issues were not trivial this time around though
15:24:58 <jrosser> anyway :) if you want to make some patches regardless that is OK
15:25:10 <noonedeadpunk> so yeah, I will check on what it would take to add EL10 support with local aio
15:25:16 <NeilHanlon> have we poked mnasiadka about the python3.11 stuff btw?
15:25:19 <noonedeadpunk> and we'll see if/how/when to land them
15:25:22 <jrosser> i was thinking to look at the usgi script thing (maybe even later today)
15:25:28 <NeilHanlon> we were chatting about NFV sig stuff this morning..
15:25:39 <jrosser> as i see that some services are migrating with no backward compatibility
15:25:48 <NeilHanlon> i'm almost at the point where like.. i'll just go rebuild all the things we need to make python3.11 work
15:25:56 <mnasiadka> What python3.11 stuff? :)
15:26:02 <noonedeadpunk> I now for fact that python3-libvirt is required by nova, and it's shipped only for 3.9
15:26:03 * NeilHanlon signing myself up for more work as always
15:26:15 <mnasiadka> Ah, you tried the same thing as me
15:26:35 <noonedeadpunk> mnasiadka: I learned that with EL8 I think
15:26:47 <opendevreview> Merged openstack/openstack-ansible stable/2025.1: Update TOX_CONSTRAINTS_FILE for stable/2025.1  https://review.opendev.org/c/openstack/openstack-ansible/+/950418
15:26:49 <noonedeadpunk> under very alike circumstances
15:26:59 <noonedeadpunk> (or maybe it was 7? who knows)
15:27:02 <NeilHanlon> Enterprise Linux: the pain that keeps on giving
15:27:13 <NeilHanlon> why do i do this to myself? lol
15:27:29 <noonedeadpunk> and I have memories of libselinux bindings for python as well, but there were some ways around...
15:27:38 <noonedeadpunk> NeilHanlon: great question, lol "D
15:28:17 <noonedeadpunk> but I'm really impressed how ppl continue shooting themselves in the leg
15:28:28 <noonedeadpunk> by building against cnetos for EL, and things like that
15:28:43 <noonedeadpunk> dropping cpu flags for still used hardware...
15:28:48 <noonedeadpunk> doh
15:29:05 <noonedeadpunk> anyway
15:29:12 <NeilHanlon> psh isn't everyone replacing their servers every 3 years to fit more pci lanes to inference AI or something?
15:29:19 <mnasiadka> Should we all stop supporting EL?:)
15:29:35 <NeilHanlon> mnasiadka: in protest? :P
15:29:57 <noonedeadpunk> NeilHanlon: and also efficiency per watt, and save costs on cooling, sure
15:30:18 <NeilHanlon> nah i mean, we all know it's been a weird situation the past few years and hasn't gotten really any clearer esp. with the CI situation and not really having any idea what's up with that all
15:30:37 <noonedeadpunk> mnasiadka: frankly? I'd internally love to do that, but I kinda realize that it's not gonna end well
15:31:08 <noonedeadpunk> internally, in terms of very personal opinion
15:32:03 <noonedeadpunk> as when trying to sell openstack, more then half are not ready to consider anything except EL
15:33:14 <noonedeadpunk> or at the very least they require the ability of rollback to EL
15:33:15 <NeilHanlon> i just had that conversation with a colleague in the rocky project on our testing team
15:33:47 <NeilHanlon> his boss wants him to investigate openstack and talk to Canonical and he is firmly planted he'd get it from RH instead
15:34:02 <NeilHanlon> i'm always trying to abduct more people into our cult :P
15:34:20 <noonedeadpunk> right :)
15:35:11 <noonedeadpunk> I especially like cases of moving from VMware due to screwed policies to RH OpenStack
15:35:21 <jrosser> mnasiadka: while you are around - did i understand right that you were looking at the rh-10-ish nodpool images?
15:35:37 <noonedeadpunk> Just as if IBM is any better then Broadcom at global scale of screwing up customers
15:36:06 <mnasiadka> Yes, I’m trying to help tonyb in CS10 work
15:37:50 <noonedeadpunk> I can try to help if there's a need
15:38:16 <noonedeadpunk> but I did not have time yet to double check the state after asking in #opendev
15:38:57 <NeilHanlon> noonedeadpunk: yeah i chose not to mention that to that colleague :P
15:39:22 <NeilHanlon> mnasiadka: and, sorry i fell off, but did you end up working on Rocky 9 nodepool images, too?
15:39:53 <mnasiadka> @NeilHanlon: yup, that’s done ;)
15:40:24 <NeilHanlon> mnasiadka: add another beer to my tab for you
15:45:27 <NeilHanlon> ok, so..
15:46:01 <NeilHanlon> short term, we drop rocky jobs and backport to the N-1(-1,-1?) releases so we can land changes
15:46:40 <NeilHanlon> at the same time, we maybe work on landing el10? I volunteer DavidGomez to help. :)
15:47:38 <NeilHanlon> noonedeadpunk: does that seem like a reasonable next steps?
15:47:38 <DavidGomez> of course, happy to help
15:48:48 <noonedeadpunk> NeilHanlon: we don't need to backport jobs removal
15:48:57 <noonedeadpunk> we'd need to backport EL10 patches
15:49:04 <NeilHanlon> ah..
15:49:04 <noonedeadpunk> ideally
15:49:07 <NeilHanlon> i had that reversed in my head
15:49:15 <NeilHanlon> well, more experience for DavidGomez! lol
15:49:33 <noonedeadpunk> lol, I won't refuse help, sure :)
15:49:39 <NeilHanlon> sorry david, you can get me back at the next onsite :P
15:50:30 <noonedeadpunk> but also adding rocky 10 image to nodepool would be amazing
15:51:33 <DavidGomez> haha not at all, sounds like a good learning opportunity
15:52:10 <noonedeadpunk> ok, cool, sounds like a plan then
15:53:51 <noonedeadpunk> #endmeeting