15:00:18 <noonedeadpunk> #startmeeting openstack_ansible_meeting 15:00:19 <opendevmeet> Meeting started Tue Jun 8 15:00:18 2021 UTC and is due to finish in 60 minutes. The chair is noonedeadpunk. Information about MeetBot at http://wiki.debian.org/MeetBot. 15:00:20 <opendevmeet> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 15:00:23 <opendevmeet> The meeting name has been set to 'openstack_ansible_meeting' 15:00:26 <noonedeadpunk> #topic rollcall 15:00:38 <noonedeadpunk> \o/ 15:00:56 <noonedeadpunk> hm, btw topic is not actually changed with meetingbot :( 15:04:51 <jrosser> hello 15:05:11 <noonedeadpunk> #topic office hours 15:05:59 <noonedeadpunk> CentOS 8.4 <- this has been solved right now I think. CI is passing again, which is awesome. But timing as usual was awful 15:06:27 <noonedeadpunk> but we also need to backport to V and I think also to U? 15:06:50 <jrosser> oh yes i expect so 15:06:56 <jrosser> thankfully they changes are small 15:07:06 <noonedeadpunk> yeah, just one symlink:) 15:07:17 <opendevreview> Dmitriy Rabotyagov proposed openstack/openstack-ansible-lxc_hosts stable/victoria: Add CentOS 8.4 support https://review.opendev.org/c/openstack/openstack-ansible-lxc_hosts/+/795282 15:07:48 <opendevreview> Dmitriy Rabotyagov proposed openstack/openstack-ansible-lxc_hosts stable/ussuri: Add CentOS 8.4 support https://review.opendev.org/c/openstack/openstack-ansible-lxc_hosts/+/795283 15:07:53 <noonedeadpunk> and thankfully 8.4 does not fail on libvirt firmware like stream still does 15:08:23 <noonedeadpunk> as https://review.opendev.org/c/openstack/openstack-ansible-os_nova/+/793007 still failing.... 15:09:30 <noonedeadpunk> So next we have 2 topics, that I expect to be merged by the end of the day 15:09:37 <noonedeadpunk> #link https://review.opendev.org/q/topic:%22osa%252Fpki%22+(status:open%20OR%20status:merged) 15:09:52 <noonedeadpunk> #link https://review.opendev.org/q/topic:%22osa%252Fbullseye%22+(status:open%20OR%20status:merged) 15:11:07 <noonedeadpunk> Once this is done, we do RC1. And probably we can branch as well, but I'd wait till next week with branching roles at least 15:11:49 <jrosser> yes - feels like all the major things are done now 15:11:50 <noonedeadpunk> we also should do bunch of rechecks... 15:13:22 <noonedeadpunk> as our gates were not really stable lately 15:14:32 <noonedeadpunk> ok, then regarding https://bugs.launchpad.net/openstack-ansible/+bug/1930276 15:14:34 <opendevmeet> Launchpad bug 1930276 in openstack-ansible "Nova API not restarted when nova policy is updated" [Undecided,Triaged] - Assigned to Dmitriy Rabotyagov (noonedeadpunk) 15:15:03 <noonedeadpunk> I'd say that it's really oslo bug. But I'm not sure if we should workaround it with triggering handlers on older branches 15:15:12 <noonedeadpunk> as it can be reproduced only with json 15:15:24 <noonedeadpunk> but it's also pretty much work I'd say.. 15:16:05 <noonedeadpunk> I wasn't able to look into oslo.policy code yet :( But will try to this week 15:16:40 <noonedeadpunk> I hope that just catching exception in correct place will do job 15:17:03 <noonedeadpunk> as otherwise we should also trigger service restart when we're removing policy.yaml 15:18:05 <spatel> noonedeadpunk jrosser i have installed aio_ovn_metal senario and seeing this - http://paste.openstack.org/show/806463/ 15:18:09 <noonedeadpunk> But I a bit doubt regarding possibility of fixing json file, since they're deprecated for a while 15:18:16 <spatel> inventory is totally empty is that ok ? 15:18:48 <spatel> sorry we are in middle of meeting. 15:20:11 <jrosser> even though the json is deprecated it's still supported in a lot of stable branchs 15:20:40 <jrosser> i'd expect there to be an interest in fixing it if we can get a good bug report 15:20:44 <noonedeadpunk> I missed oslo meeting yestarday to raise this :( 15:22:03 <noonedeadpunk> fair ppint 15:22:06 <noonedeadpunk> *point 15:22:15 <jrosser> lets ask them about our bug 15:23:50 <noonedeadpunk> I was asking but haven't pinged hberaud so get no answer:) 15:24:16 <jrosser> ah well i just did, lets see if anything happens 15:25:26 <noonedeadpunk> I guess they mostly following on channel during meetings 15:26:30 <noonedeadpunk> regarding https://specs.openstack.org/openstack/openstack-ansible-specs/specs/xena/protecting-plaintext-configs.html - I think it's worth creating new repo for vault only after branching 15:27:25 <noonedeadpunk> I saw folks have been asking here about backporting this work to previous branches, which is kind of important point for them. But I'd say that's not possible considering amount of work and requirement of vault role 15:29:17 <jrosser> yes it looks awkward 15:29:34 <noonedeadpunk> Oh! And regarding DDoS! What I think we can do - in python_venv_build, check for amount of hosts in the play and fail, in case there're more then let's say 5 hosts, and wheels won't be built for $reasons 15:29:56 <jrosser> yes like safety net of some sort 15:31:09 <noonedeadpunk> and maybe link to some doc about troubleshooting this? And suggest workaround of setting osa_serial to lower then 5 if it can't be fixed for any reason 15:31:14 <jrosser> we can probably print out useful stuff in a fail: message about number of hosts, number of repo servers and stuff 15:31:26 <noonedeadpunk> yeah, totally 15:31:32 <jrosser> OS releases / architecture etc, to show why it's not worked 15:32:09 <noonedeadpunk> yeah, really good idea. it would be easier to understand for deployer what he does wrong 15:32:45 <noonedeadpunk> Also I was thinking of moving https://docs.openstack.org/openstack-ansible/rocky/admin/upgrades/distribution-upgrades.html to master as well in regards to focal upgrade? 15:33:06 <noonedeadpunk> and do some critical note about that you can't just upgrade computes without having repo container 15:33:38 <noonedeadpunk> but yeah, having repo container might be really tricky. Smth we probably should figure out in terms of the best way to sort this out 15:33:52 <jrosser> ah - does that page not appear in all branches 15:33:56 <noonedeadpunk> nope 15:34:00 <noonedeadpunk> it's only for rocky 15:34:11 <noonedeadpunk> it has been merged directly there 15:34:28 <jrosser> right - rocky was xenial->bionic, but i guess that W is a really huge transition release for all OS 15:34:36 <noonedeadpunk> yep 15:35:04 <noonedeadpunk> well, for centos it would be relatively easy migration, but debian and ubuntu... 15:35:42 <jrosser> theres a similar thing to bring arm nodes into an existing x86_64 cloud 15:35:51 <jrosser> nothing prevents that other than missing a repo server 15:36:00 <noonedeadpunk> eventually we have planned bionic->focal upgrade for autumn, so it will be adjusted anyway, but would be good to look into it in advance 15:36:26 <noonedeadpunk> and the biggest thing here is actually lsync 15:36:36 <jrosser> i feel like we maybe miss something here to bring in temporary / alternative repo hosts for specific cases 15:37:30 <noonedeadpunk> you mean for being standalone (ie under proxy)? 15:38:16 <jrosser> right, it's easy to point the build host at something else for an ansible group 15:38:29 <jrosser> but not so easy to get the build results to appear at the LB 15:38:52 <noonedeadpunk> ah 15:39:08 <noonedeadpunk> I think we can leverage ACLs or smth like that 15:39:17 <noonedeadpunk> an be a bit smarter here... 15:39:53 <noonedeadpunk> actually nothing stopps us from having several group of backends 15:40:06 <jrosser> oh well actually i wonder if overriding openstack_repo_url for a host group would work 15:41:15 <noonedeadpunk> it probably should... but it's not that neat.. As you need to set it manually and define extra group for new OS 15:41:50 <noonedeadpunk> and well, if you have more then 1 repo container per specific case... 15:42:06 <noonedeadpunk> that would be not handy really 15:44:17 <noonedeadpunk> oh, btw. I saw really weird failures during facts gathering after switching to min 15:45:25 <jrosser> oh dear - anything specific? 15:45:26 <noonedeadpunk> https://700a7e3ec90813bffe3e-a140c73e010d1e58296165dc52d0a2c3.ssl.cf2.rackcdn.com/788031/11/check/openstack-ansible-deploy-aio_lxc-debian-buster/c9ca1d5/logs/ara-report/results/3366.html 15:45:48 <noonedeadpunk> And I saw it periodicly on different patches 15:46:10 <noonedeadpunk> no details in stdout either 15:47:19 <noonedeadpunk> so have literally no idea wtf is going on... 15:48:46 <dmsimard> those are the worst 15:49:06 <dmsimard> sometimes you can get the actual traceback by adding -vvv 15:49:09 <noonedeadpunk> dmsimard: have you seen smth like that in other places? 15:49:20 <dmsimard> ansible eats tracebacks unless it's -vvv 15:49:43 <noonedeadpunk> yeah... but you need to catch it first... and until it's random and not really reproducable... 15:50:04 <noonedeadpunk> at least I couldn't reproduce that relatively easily... 15:50:43 <dmsimard> it's delegating the setup task that fails ? maybe one of the item from the list doesn't work for some reason 15:51:54 <noonedeadpunk> well, it's a container on the same host... and in case of CI item consist of single element 15:52:10 <noonedeadpunk> but that's actually what I also thought about 15:54:55 <noonedeadpunk> I could imagine this happening in case of OOM but haven't found any proof in the log 15:57:43 <jrosser> not found anything suspicious at all here 15:59:36 <noonedeadpunk> but I also don't see it happening today... 16:01:01 <noonedeadpunk> #endmeeting