14:59:39 <noonedeadpunk> #startmeeting openstack_ansible_meeting 14:59:39 <opendevmeet> Meeting started Tue Oct 3 14:59:39 2023 UTC and is due to finish in 60 minutes. The chair is noonedeadpunk. Information about MeetBot at http://wiki.debian.org/MeetBot. 14:59:39 <opendevmeet> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 14:59:39 <opendevmeet> The meeting name has been set to 'openstack_ansible_meeting' 14:59:45 * NeilHanlon needs more coffee 14:59:48 <noonedeadpunk> #topic rollcall 14:59:55 <noonedeadpunk> Yeah, apparently me too :D 14:59:58 <NeilHanlon> hehe 15:00:02 <NeilHanlon> we're just a bit early today 15:00:36 <mgariepy> hey ! 15:01:01 <noonedeadpunk> \o/ 15:01:06 <jrosser> o/ hello 15:01:15 <damiandabrowski> hi! 15:01:28 <noonedeadpunk> #topic bug triage 15:01:30 <noonedeadpunk> To be frank - I in fact don't have much updates/to say from the last meeting. Was mostly occupied with internal things 15:01:44 <noonedeadpunk> But, I've submitted couple of new bugs we've found during the upgrade 15:02:53 <noonedeadpunk> But lets start from this one 15:02:56 <noonedeadpunk> #link https://bugs.launchpad.net/openstack-ansible/+bug/2036421 15:03:17 <noonedeadpunk> I;ve replied to it, but I'm not sure if we wanna do smth there or just mark as invalid 15:03:30 <openstack-ansib> I want to see if Horizon has been translated into Farsi or not?! 15:03:34 <noonedeadpunk> As I see how this release note can be confusing, but I'm not sure what can be done 15:04:18 <noonedeadpunk> openstack-ansib: you can see horizon translations here: https://translate.openstack.org/iteration/view/horizon/master?dswid=7334 15:05:14 <noonedeadpunk> regarding bug - I'm not sure what we can do at this point... We could add another release note back in the days, but not I guess it's useless 15:05:20 <noonedeadpunk> *now 15:06:40 <noonedeadpunk> So I guess I will mark it as invalid if no objections 15:07:02 <NeilHanlon> that seems reasonable 15:08:29 <noonedeadpunk> #link https://bugs.launchpad.net/openstack-ansible/+bug/2037554 15:08:35 <noonedeadpunk> this one is actually really bad 15:08:46 <noonedeadpunk> and it's not limited to what is described there... 15:09:07 <noonedeadpunk> I've catched another phase of this bug yesterday 15:10:05 <jrosser> should we make that a separate play? 15:10:05 <noonedeadpunk> So it failed on us when we were deploying new compute node 15:10:13 <jrosser> just to make it simple 15:10:31 <noonedeadpunk> As it attempted to do db migration against compute 15:10:40 <noonedeadpunk> jrosser: yeah, I was thinking exactly about that tbh 15:11:08 <noonedeadpunk> except we have quite some complexity in the playbook for nova... I guess that stopped me from doing this originally 15:12:41 <noonedeadpunk> as I was thinking to add some post_tasks or do tasks_from, but that doesn't look too trivial with how we do things here https://opendev.org/openstack/openstack-ansible/src/branch/master/playbooks/os-nova-install.yml#L54-L58 15:12:43 <ayush> There is another bug here 15:12:45 <ayush> https://bugs.launchpad.net/osops/+bug/2038315 15:13:26 <noonedeadpunk> Ok, moved it to the correct project 15:14:22 <jrosser> common-playbooks/nova.yml is kind of nasty too 15:14:24 <noonedeadpunk> sorry for not describing bugs in a good way - was pretty much in a rush and needed not to forget about them 15:14:42 <jrosser> as it attempts to be common, then a whole bunch of the tasks are conditional for computes only 15:14:46 <noonedeadpunk> yup, it pretty much is 15:14:47 <jrosser> so that could be simplified too 15:14:57 <jrosser> like too much optimisation there makes a mess 15:15:09 <noonedeadpunk> I wonder if we need to have that common playbook 15:15:18 <jrosser> true 15:15:32 <jrosser> perhaps some tidying here would make a neat place to deal with that bug 15:15:36 <noonedeadpunk> as it may be way more plain when moved to os-nova-install 15:15:50 <noonedeadpunk> ok, sounds like a plan then 15:16:17 <noonedeadpunk> hopefully, it will be backportable enough... 15:16:23 <noonedeadpunk> #link https://bugs.launchpad.net/openstack-ansible/+bug/2038315 15:16:40 <noonedeadpunk> So, we have some stable branches borked due to ansible.netcommon breaking galaxy 15:16:55 <noonedeadpunk> I didn't have time to look there today, but we're catching failures internally as well 15:17:01 <noonedeadpunk> Except 2023.1 - it's working fine 15:17:12 <noonedeadpunk> (as we moved to github from galaxy there) 15:17:24 <noonedeadpunk> ayush: did workaround I've posted you helped? 15:17:46 <noonedeadpunk> ie - create user-collection-requirements.yml with https://paste.openstack.org/show/bC5oFVEOBoOdB9yMb9Dg/ ? 15:18:14 <NeilHanlon> i can probably find some time to look at the galaxy problem today/tomorrow 15:18:17 <noonedeadpunk> damiandabrowski: actually, you've catched that yesterday as well 15:18:42 <jrosser> what actually is the bug? 15:18:55 <noonedeadpunk> that bootstrap fails of some stable branches 15:19:12 <jrosser> oh sure but is there a new netcommon that breaks our old stuff? 15:19:15 <noonedeadpunk> because it can't install ansible.netcommon for $reasons 15:19:20 <jrosser> ahha ok 15:19:30 <noonedeadpunk> yeah, I believe it just breaks galaxy overall 15:20:00 <noonedeadpunk> I believe that's related to them having `v` in tags, while not having in versions or smth like that 15:20:14 <noonedeadpunk> (as ansible wasn't happy either in some scenarios) 15:20:31 <NeilHanlon> "unexpected error when getting available versions of collection" 15:20:38 <NeilHanlon> such a helpful message 🙃 15:20:46 <noonedeadpunk> Skipping Galaxy server https://galaxy.ansible.com/api/. Got an unexpected error when getting available versions of collection ansible.netcommon 15:20:47 <noonedeadpunk> yeah 15:21:18 <NeilHanlon> #link https://github.com/ansible-collections/ansible.netcommon/issues/562 15:21:24 <ayush> @noonedeadpunk : i have not tried, i will try and let you know the results by tmrw 15:21:29 <ayush> mrng 15:22:12 <noonedeadpunk> But, I see no new commits in collection 15:22:21 <noonedeadpunk> And web ui of galaxy doesn't seem broken 15:22:26 <noonedeadpunk> https://galaxy.ansible.com/ui/repo/published/ansible/netcommon/ 15:22:45 <noonedeadpunk> So... it's either already fixed... Or API != UI 15:22:50 <NeilHanlon> ope 15:22:51 <NeilHanlon> # https://github.com/ansible/awx/issues/14496 15:22:56 <jrosser> this is "galaxy ng" isnt it? 15:22:57 <NeilHanlon> #link https://github.com/ansible/awx/issues/14496 15:23:13 <noonedeadpunk> "We've had similiar issue today but with 'ansible.netcommon' collection" 15:23:20 <noonedeadpunk> " Removal of default Galaxy credential from organisation settings helped us mitigate the problem." 15:23:24 <NeilHanlon> > However the old-galaxy.ansible.com settings seems to have solved the issue for us ;-) 15:23:40 <noonedeadpunk> Lol 15:23:53 * NeilHanlon adds liquor to his coffee 15:23:59 <noonedeadpunk> This reminds me of some company.... that maintains CentOS Stream... 15:24:08 <noonedeadpunk> What the name was that... 15:24:12 <noonedeadpunk> Anyway 15:24:38 <noonedeadpunk> Might be we can replace galaxy with github or just wait for the fix... 15:25:17 <noonedeadpunk> OR, we need to update ansible version... 15:25:24 <NeilHanlon> i'm gonna ping greg sutcliffe.. get his thoughts :) 15:26:52 <noonedeadpunk> Yeah, would be nice to hear 15:26:57 <NeilHanlon> seems like new ansible won't help 15:27:05 <noonedeadpunk> #link https://bugs.launchpad.net/openstack-ansible/+bug/2037649 15:27:08 <NeilHanlon> #link https://forum.ansible.com/t/awx-cannot-download-collections-roles-from-the-new-galaxy/1251/4?u=neil 15:27:52 <noonedeadpunk> that looks really messy... 15:28:11 <noonedeadpunk> What makes it worse, that in branches like Xena we also had issues with rabbitmq versions... 15:28:19 <NeilHanlon> :( 15:28:19 <NeilHanlon> yeah 15:29:09 <noonedeadpunk> so 2037649 is another race condition bug, but thankfully it affects only vpnaas 15:29:09 <mgariepy> rabbitmq is a mess.. 15:29:29 <noonedeadpunk> jamesdenton: you might be interested in https://bugs.launchpad.net/openstack-ansible/+bug/2037649 as well 15:29:58 <noonedeadpunk> I will check what's the best approach, but I guess it would be change of the destination directory to the one inside the venv 15:30:10 <noonedeadpunk> so that symlinking would be not a problem 15:30:18 <noonedeadpunk> #topic office hours 15:30:34 <noonedeadpunk> I failed to book a PTG slots until now :( 15:30:42 <noonedeadpunk> Will try to do that during this week 15:31:11 <noonedeadpunk> Another thing is that we likely should switch from master to tracking stable/2023.2 since all projects have branched already 15:31:40 <noonedeadpunk> Will propose patches soonish 15:31:48 <noonedeadpunk> #action noonedeadpunk book PTG slots 15:32:03 <jrosser> do we have a TODO list for bobcat 15:32:03 <noonedeadpunk> #action noonedeadpunk propose patch to track 2023.2 15:32:10 <jrosser> like actual patches to chase and deal with 15:32:19 <jrosser> seems to be many many open 15:32:54 <jrosser> i think we also need to give the CI some tlc 15:33:11 <jrosser> jobs seem to be running slowly, upgrades are TIMEOUT regularly atm 15:33:32 <noonedeadpunk> I think it's Debian 12, quorum queues at minimum 15:33:39 <jrosser> then there is a fair number of failing to retrieve u-c with https 15:33:39 <noonedeadpunk> we can discuss rest, like Skyline 15:34:15 <jrosser> and I see a number of Rocky repo errors every day still 15:34:33 <noonedeadpunk> But what we tried to scope at the beginning of cycle is here 15:34:35 <noonedeadpunk> #link https://etherpad.opendev.org/p/osa-bobcat-ptg 15:35:02 <noonedeadpunk> Ah, we have openstack-resources role 15:35:26 <noonedeadpunk> #link https://review.opendev.org/q/topic:osa%252Fopenstack_resources 15:35:58 <noonedeadpunk> Ugh, it's failing with same galaxy issue 15:36:33 <NeilHanlon> bleh 15:36:36 <jrosser> oh no curse of the tests repo 15:36:50 <noonedeadpunk> yeah-yeah... 15:37:01 <NeilHanlon> I will look into rocky issues, jrosser. i know of a few lingering problems I'm trying to snuff out.. but also wondering based on what you're saying and I'm seeing if I just need to give the CDN a good ol' flush. 15:37:06 <noonedeadpunk> and modular libvirt would be also a thing nice to have 15:37:13 <NeilHanlon> yea, i have an update on that 15:37:28 <NeilHanlon> It's basically confirmed to be the only thing available in EL10 15:37:50 <NeilHanlon> and ELN composes have begun for what will become CentOS Stream 10 15:37:58 <NeilHanlon> https://composes.stream.centos.org/stream-10/ 15:38:31 <noonedeadpunk> so we need to get to bottom of that sooner or later anyway 15:38:49 <noonedeadpunk> And it seems that Debian is not going the same path, at least not with Debian 12? 15:40:32 <NeilHanlon> yeah. i am going to schedule myself some time to come up with a plan, especially around how to migrate/upgrade to modular libvirt 15:41:38 <noonedeadpunk> well. if it's only EL10... 15:41:57 <noonedeadpunk> Maybe no migration needed - you still can't upgrade 9->10 15:42:08 <noonedeadpunk> and then we can wait for 10 for that 15:42:10 <NeilHanlon> true, but i wonder about folks running mixed environments? 15:42:21 <noonedeadpunk> make up some conditions? 15:42:23 <NeilHanlon> i.e., 9 and 10 15:42:41 <NeilHanlon> maybe you're right--we can just not worry about it until we support EL10 15:42:43 <noonedeadpunk> like define services to restart in vars per distro 15:42:54 <NeilHanlon> yeah 15:43:06 <noonedeadpunk> like we have already here: https://opendev.org/openstack/openstack-ansible-os_nova/src/branch/master/vars/redhat.yml#L19-L21 15:43:15 <NeilHanlon> right right 15:43:25 <noonedeadpunk> But I bet it would be way-way more complicated.... 15:43:58 <noonedeadpunk> As I'm not sure I understand logic behind this modular thing and how to make tls/non-tls things to work... 15:44:10 <noonedeadpunk> And the order to start/restart 15:45:13 <NeilHanlon> yeah 15:45:23 <NeilHanlon> i'm not looking forward to it, honestly. lol 15:48:13 <noonedeadpunk> so, does the list of Debian 12, quorum queues support, and openstack-resource role (possibly) sounds good enough? 15:48:35 <noonedeadpunk> I was planning also to work on PKI improvement to add vault support, but I'm not sure if I will be able to fit 15:48:46 <noonedeadpunk> And also would better release earlier... 15:50:06 <NeilHanlon> that sounds good to me 15:50:13 <NeilHanlon> would rather less things we're more confident with 15:50:57 <noonedeadpunk> I will also try to work on documentation review. I've already posted some changes, inlcuding some OVN drawing here https://review.opendev.org/c/openstack/openstack-ansible/+/894384 15:51:07 <noonedeadpunk> which would be great to get some feedbacks on... 15:51:24 <noonedeadpunk> And was going to make openstack_user_config examples OVN-friendly 15:51:41 <noonedeadpunk> As I guess most users copying them and running into issues afterwards 15:53:10 <noonedeadpunk> ok, so if that's it 15:53:17 <noonedeadpunk> #endmeeting