15:00:18 <noonedeadpunk> #startmeeting openstack_ansible_meeting
15:00:19 <opendevmeet> Meeting started Tue Jun  8 15:00:18 2021 UTC and is due to finish in 60 minutes.  The chair is noonedeadpunk. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:00:20 <opendevmeet> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
15:00:23 <opendevmeet> The meeting name has been set to 'openstack_ansible_meeting'
15:00:26 <noonedeadpunk> #topic rollcall
15:00:38 <noonedeadpunk> \o/
15:00:56 <noonedeadpunk> hm, btw topic is not actually changed with meetingbot :(
15:04:51 <jrosser> hello
15:05:11 <noonedeadpunk> #topic office hours
15:05:59 <noonedeadpunk> CentOS 8.4 <- this has been solved right now I think. CI is passing again, which is awesome. But timing as usual was awful
15:06:27 <noonedeadpunk> but we also need to backport to V and I think also to U?
15:06:50 <jrosser> oh yes i expect so
15:06:56 <jrosser> thankfully they changes are small
15:07:06 <noonedeadpunk> yeah, just one symlink:)
15:07:17 <opendevreview> Dmitriy Rabotyagov proposed openstack/openstack-ansible-lxc_hosts stable/victoria: Add CentOS 8.4 support  https://review.opendev.org/c/openstack/openstack-ansible-lxc_hosts/+/795282
15:07:48 <opendevreview> Dmitriy Rabotyagov proposed openstack/openstack-ansible-lxc_hosts stable/ussuri: Add CentOS 8.4 support  https://review.opendev.org/c/openstack/openstack-ansible-lxc_hosts/+/795283
15:07:53 <noonedeadpunk> and thankfully 8.4 does not fail on libvirt firmware like stream still does
15:08:23 <noonedeadpunk> as https://review.opendev.org/c/openstack/openstack-ansible-os_nova/+/793007 still failing....
15:09:30 <noonedeadpunk> So next we have 2 topics, that I expect to be merged by the end of the day
15:09:37 <noonedeadpunk> #link https://review.opendev.org/q/topic:%22osa%252Fpki%22+(status:open%20OR%20status:merged)
15:09:52 <noonedeadpunk> #link https://review.opendev.org/q/topic:%22osa%252Fbullseye%22+(status:open%20OR%20status:merged)
15:11:07 <noonedeadpunk> Once this is done, we do RC1. And probably we can branch as well, but I'd wait till next week with branching roles at least
15:11:49 <jrosser> yes - feels like all the major things are done now
15:11:50 <noonedeadpunk> we also should do bunch of rechecks...
15:13:22 <noonedeadpunk> as our gates were not really stable lately
15:14:32 <noonedeadpunk> ok, then regarding https://bugs.launchpad.net/openstack-ansible/+bug/1930276
15:14:34 <opendevmeet> Launchpad bug 1930276 in openstack-ansible "Nova API not restarted when nova policy is updated" [Undecided,Triaged] - Assigned to Dmitriy Rabotyagov (noonedeadpunk)
15:15:03 <noonedeadpunk> I'd say that it's really oslo bug. But I'm not sure if we should workaround it with triggering handlers on older branches
15:15:12 <noonedeadpunk> as it can be reproduced only with json
15:15:24 <noonedeadpunk> but it's also pretty much work I'd say..
15:16:05 <noonedeadpunk> I wasn't able to look into oslo.policy code yet :( But will try to this week
15:16:40 <noonedeadpunk> I hope that just catching exception in correct place will do job
15:17:03 <noonedeadpunk> as otherwise we should also trigger service restart when we're removing policy.yaml
15:18:05 <spatel> noonedeadpunk jrosser i have installed aio_ovn_metal senario and seeing this - http://paste.openstack.org/show/806463/
15:18:09 <noonedeadpunk> But I a bit doubt regarding possibility of fixing json file, since they're deprecated for a while
15:18:16 <spatel> inventory is totally empty is that ok ?
15:18:48 <spatel> sorry we are in middle of meeting.
15:20:11 <jrosser> even though the json is deprecated it's still supported in a lot of stable branchs
15:20:40 <jrosser> i'd expect there to be an interest in fixing it if we can get a good bug report
15:20:44 <noonedeadpunk> I missed oslo meeting yestarday to raise this :(
15:22:03 <noonedeadpunk> fair ppint
15:22:06 <noonedeadpunk> *point
15:22:15 <jrosser> lets ask them about our bug
15:23:50 <noonedeadpunk> I was asking but haven't pinged hberaud so get no answer:)
15:24:16 <jrosser> ah well i just did, lets see if anything happens
15:25:26 <noonedeadpunk> I guess they mostly following on channel during meetings
15:26:30 <noonedeadpunk> regarding https://specs.openstack.org/openstack/openstack-ansible-specs/specs/xena/protecting-plaintext-configs.html - I think it's worth creating new repo for vault only after branching
15:27:25 <noonedeadpunk> I saw folks have been asking here about backporting this work to previous branches, which is kind of important point for them. But I'd say that's not possible considering amount of work and requirement of vault role
15:29:17 <jrosser> yes it looks awkward
15:29:34 <noonedeadpunk> Oh! And regarding DDoS! What I think we can do - in python_venv_build, check for amount of hosts in the play and fail, in case there're more then let's say 5 hosts, and wheels won't be built for $reasons
15:29:56 <jrosser> yes like safety net of some sort
15:31:09 <noonedeadpunk> and maybe link to some doc about troubleshooting this? And suggest workaround of setting osa_serial to lower then 5 if it can't be fixed for any reason
15:31:14 <jrosser> we can probably print out useful stuff in a fail: message about number of hosts, number of repo servers and stuff
15:31:26 <noonedeadpunk> yeah, totally
15:31:32 <jrosser> OS releases / architecture etc, to show why it's not worked
15:32:09 <noonedeadpunk> yeah, really good idea. it would be easier to understand for deployer what he does wrong
15:32:45 <noonedeadpunk> Also I was thinking of moving https://docs.openstack.org/openstack-ansible/rocky/admin/upgrades/distribution-upgrades.html to master as well in regards to focal upgrade?
15:33:06 <noonedeadpunk> and do some critical note about that you can't just upgrade computes without having repo container
15:33:38 <noonedeadpunk> but yeah, having repo container might be really tricky. Smth we probably should figure out in terms of the best way to sort this out
15:33:52 <jrosser> ah - does that page not appear in all branches
15:33:56 <noonedeadpunk> nope
15:34:00 <noonedeadpunk> it's only for rocky
15:34:11 <noonedeadpunk> it has been merged directly there
15:34:28 <jrosser> right - rocky was xenial->bionic, but i guess that W is a really huge transition release for all OS
15:34:36 <noonedeadpunk> yep
15:35:04 <noonedeadpunk> well, for centos it would be relatively easy migration, but debian and ubuntu...
15:35:42 <jrosser> theres a similar thing to bring arm nodes into an existing x86_64 cloud
15:35:51 <jrosser> nothing prevents that other than missing a repo server
15:36:00 <noonedeadpunk> eventually we have planned bionic->focal upgrade for autumn, so it will be adjusted anyway, but would be good to look into it in advance
15:36:26 <noonedeadpunk> and the biggest thing here is actually lsync
15:36:36 <jrosser> i feel like we maybe miss something here to bring in temporary / alternative repo hosts for specific cases
15:37:30 <noonedeadpunk> you mean for being standalone (ie under proxy)?
15:38:16 <jrosser> right, it's easy to point the build host at something else for an ansible group
15:38:29 <jrosser> but not so easy to get the build results to appear at the LB
15:38:52 <noonedeadpunk> ah
15:39:08 <noonedeadpunk> I think we can leverage ACLs or smth like that
15:39:17 <noonedeadpunk> an be a bit smarter here...
15:39:53 <noonedeadpunk> actually nothing stopps us from having several group of backends
15:40:06 <jrosser> oh well actually i wonder if overriding openstack_repo_url for a host group would work
15:41:15 <noonedeadpunk> it probably should... but it's not that neat.. As you need to set it manually and define extra group for new OS
15:41:50 <noonedeadpunk> and well, if you have more then 1 repo container per specific case...
15:42:06 <noonedeadpunk> that would be not handy really
15:44:17 <noonedeadpunk> oh, btw. I saw really weird failures during facts gathering after switching to min
15:45:25 <jrosser> oh dear - anything specific?
15:45:26 <noonedeadpunk> https://700a7e3ec90813bffe3e-a140c73e010d1e58296165dc52d0a2c3.ssl.cf2.rackcdn.com/788031/11/check/openstack-ansible-deploy-aio_lxc-debian-buster/c9ca1d5/logs/ara-report/results/3366.html
15:45:48 <noonedeadpunk> And I saw it periodicly on different patches
15:46:10 <noonedeadpunk> no details in stdout either
15:47:19 <noonedeadpunk> so have literally no idea wtf is going on...
15:48:46 <dmsimard> those are the worst
15:49:06 <dmsimard> sometimes you can get the actual traceback by adding -vvv
15:49:09 <noonedeadpunk> dmsimard: have you seen smth like that in other places?
15:49:20 <dmsimard> ansible eats tracebacks unless it's -vvv
15:49:43 <noonedeadpunk> yeah... but you need to catch it first... and until it's random and not really reproducable...
15:50:04 <noonedeadpunk> at least I couldn't reproduce that relatively easily...
15:50:43 <dmsimard> it's delegating the setup task that fails ? maybe one of the item from the list doesn't work for some reason
15:51:54 <noonedeadpunk> well, it's a container on the same host... and in case of CI item consist of single element
15:52:10 <noonedeadpunk> but that's actually what I also thought about
15:54:55 <noonedeadpunk> I could imagine this happening in case of OOM but haven't found any proof in the log
15:57:43 <jrosser> not found anything suspicious at all here
15:59:36 <noonedeadpunk> but I also don't see it happening today...
16:01:01 <noonedeadpunk> #endmeeting