15:01:27 <noonedeadpunk> #startmeeting openstack_ansible_meeting
15:01:27 <opendevmeet> Meeting started Tue Nov 28 15:01:27 2023 UTC and is due to finish in 60 minutes.  The chair is noonedeadpunk. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:01:27 <opendevmeet> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
15:01:27 <opendevmeet> The meeting name has been set to 'openstack_ansible_meeting'
15:01:34 <noonedeadpunk> #topic rollcall
15:02:06 <noonedeadpunk> o/
15:10:05 <noonedeadpunk> #topic office hours
15:10:31 <noonedeadpunk> So, given we're really on a deadline for 2023.2, I think we should leave openstack_resources for the next cycle
15:10:48 <noonedeadpunk> And I will propose roles branching soonish to meet the deadline
15:11:14 <noonedeadpunk> I've looked briefly through outstanding patches, and things look okeyish
15:12:03 <noonedeadpunk> The most critical thing (by far) are for integrated repo
15:12:25 <noonedeadpunk> basically this: https://review.opendev.org/c/openstack/openstack-ansible/+/901204
15:13:21 <NeilHanlon> I agree re: openstack_resources if we still need some time to iterate on it. would rather have it finalized more or less
15:13:53 <noonedeadpunk> I also start seeing more rocky failures lately fwiw...
15:14:11 <noonedeadpunk> With things like before: `"Failed to download metadata for repo 'baseos': Yum repo downloading error: Downloading error(s): repodata/0e0e0252-80d4-47bf-bb11-1f97d7aac5b0-PRIMARY.xml.gz - Cannot download, all mirrors were already tried without success`
15:14:18 <noonedeadpunk> example: https://zuul.opendev.org/t/openstack/build/80791ccb907f47a2889e8b65c3abc367
15:14:52 <noonedeadpunk> NeilHanlon: had you any luck talking to infra ppl about mirrors?
15:17:30 <NeilHanlon> yea, mnasiadka started the conversation and I need to follow back up with them. it seems like it's probably OK but they need some more info around the mirror source, how to verify they're up to date, etc
15:18:10 <noonedeadpunk> ok, I see
15:19:44 <noonedeadpunk> We had actually couple of bugs reported on launchpad.
15:19:58 <noonedeadpunk> #link https://bugs.launchpad.net/openstack-ansible/+bug/2044629
15:20:04 <noonedeadpunk> this one I was not able to reproduce.
15:20:30 <noonedeadpunk> Also I believe we should be having a job now that would catch invalid SHAs.
15:20:54 <noonedeadpunk> ANd also, as Neil rightfully stated, Rocky 8 for 2023.1 is not a good combination anyway
15:21:56 <noonedeadpunk> #link https://bugs.launchpad.net/openstack-ansible/+bug/2038818
15:22:40 <mnasiadka> Rocky 8? That's rather Yoga material ;-)
15:22:43 <noonedeadpunk> I was looking through it shortly, but I'm actually not sure what we can/should do at this point
15:22:56 <noonedeadpunk> mnasiadka: it exactly is!
15:23:48 <noonedeadpunk> But I was considering it only because issue was related to `git clone` basically, which should not differ much
15:24:38 <noonedeadpunk> andrewbonney: do you have anyy suggestions for 2038818?
15:25:03 <andrewbonney> Not really. I couldn't see an easy option to work out which flags to switch on ahead of upgrade
15:25:32 <andrewbonney> If we supported rolling upgrades it would be easier to avoid breaking things inadvertently, but that doesn't help with identifying the flags to switch on
15:25:55 <noonedeadpunk> Well. We've enabled maintenance flag and used it couple of times.
15:26:05 <noonedeadpunk> I can't say that helped....
15:26:38 <noonedeadpunk> As each time node was returning from maintenance consumers were still crashing and not always recovering
15:29:04 <noonedeadpunk> So use maintenance and "rolling" upgrades still with caution ;)
15:29:20 <noonedeadpunk> I had reported one bug as well btw
15:29:33 <noonedeadpunk> #link https://bugs.launchpad.net/openstack-ansible/+bug/2044229
15:30:01 <noonedeadpunk> where basically, from some point in time, our connection plugin does not respect remote_user provided
15:30:11 <noonedeadpunk> but ansible_user is being respected
15:30:23 <noonedeadpunk> I didn't have to look into more details yet, but will do so
15:31:09 <jrosser> i was sure there was some stuff in the tests repo to check user stuff with the connection plugin
15:31:17 <jrosser> though perhaps that was more related to become:
15:31:37 <noonedeadpunk> yes, become works as well
15:31:56 <noonedeadpunk> what doesn't work basically is `openstack-ansible -u ubuntu`
15:32:04 <noonedeadpunk> it's always end up as root
15:32:16 <noonedeadpunk> but `openstack-ansible -e ansible_user=ubuntu` would work
15:32:30 <noonedeadpunk> which was slightly /o\
15:34:08 <noonedeadpunk> As very confusing also to see correct user in ouput right before connection command, but not in the SSH itself
15:34:33 <jrosser> NeilHanlon: would a rocky mirror actually help with `Failed to download metadata for repo 'baseos'`
15:36:45 <NeilHanlon> i think it should, yes. the 'problem' so to speak is that the default configuration for our repos uses the mirrorlist URL, which can go to, more or less, anywhere (it's Geo-based, but, will fill in). so hard coding that to anything (even our canonical dl.rockylinux.org) should mitigate some percentage of the spurious failures
15:37:41 <noonedeadpunk> So maybe we should do that then?
15:38:07 <jrosser> is there an interim way we can do that in the zuul pre job?
15:38:08 <opendevreview> Merged openstack/openstack-ansible-plugins master: Add no_log to setup_roles inlcude  https://review.opendev.org/c/openstack/openstack-ansible-plugins/+/901755
15:39:15 <noonedeadpunk> there;s some preped nodepool file that we consume iirc...
15:39:44 <opendevreview> Dmitriy Rabotyagov proposed openstack/openstack-ansible-plugins stable/2023.1: Add no_log to setup_roles inlcude  https://review.opendev.org/c/openstack/openstack-ansible-plugins/+/902068
15:41:17 <jrosser> https://github.com/openstack/openstack-ansible/blob/master/tests/roles/bootstrap-host/tasks/prepare_aio_config.yml#L77-L92
15:41:26 <noonedeadpunk> `/etc/ci/mirror_info.sh`
15:42:28 <noonedeadpunk> yeah that actually works as well
15:42:44 <jrosser> here we grab it https://github.com/openstack/openstack-ansible/blob/master/tests/roles/bootstrap-host/tasks/gather_nodepool_vars.yml#L27
15:43:22 <noonedeadpunk> just found it as well
15:43:41 <noonedeadpunk> So I guess we can add some logic for rocky specifically
15:43:53 <NeilHanlon> the 'easy' thing is to just search/replace /etc/yum.repos.d/*.repo and replace `^mirrorlist` with `#mirrorlist` and `^#baseurl` with `baseurl`
15:44:26 <NeilHanlon> https://git.resf.org/sig_cloud/images/src/branch/main/elements/rocky-minimal-yum/pre-install.d/03-yum-cleanup#L39 (a new thing I've been messing around with 👀
15:45:42 <noonedeadpunk> I'm actually confused about openstack_hosts_centos_mirror_url
15:45:49 <noonedeadpunk> as well as centos_mirror_url
15:45:56 <noonedeadpunk> as they're actually never used today?
15:47:04 <noonedeadpunk> and eventually we don't touch default mirrors provided/configured for the image
15:47:15 <NeilHanlon> 🤔
15:47:30 <NeilHanlon> yea i don't really see it used either
15:48:03 <jrosser> that really is useless vars
15:49:14 <noonedeadpunk> uca_apt_repo_url was jsut renamed to apt_repo_url
15:49:20 <noonedeadpunk> https://opendev.org/openstack/openstack-ansible-openstack_hosts/src/branch/master/releasenotes/notes/apt_repo-097cb34c12e60207.yaml
15:49:46 <noonedeadpunk> probably openstack_hosts_centos_mirror_url was also for RDO specifically?
15:50:07 <noonedeadpunk> but yes, that's need some love I believe...
15:50:34 <jrosser> shame no-one tries "difficult" installed (like limited connectivity) with RH variants
15:50:46 <NeilHanlon> i will poke at a zuul override for rocky using this, and stack it with a cleanup of that cruft
15:50:46 <jrosser> becasue this surfaces all sorts of corner cases
15:50:52 <NeilHanlon> agreed jrosser
15:51:54 <NeilHanlon> we have also been considering releasing a package which provides an easy way to manipulate repo configs, as we often have our testing team need to perform this very operation (change mirrorlist->baseurl and modify contentdir) and it gets... boring
15:57:04 <noonedeadpunk> yeah, totally understandable. but package sounds... slightly excessive?:))
15:58:35 <noonedeadpunk> #endmeeting