15:00:24 <noonedeadpunk> #startmeeting openstack_ansible_meeting
15:00:24 <opendevmeet> Meeting started Tue May 24 15:00:24 2022 UTC and is due to finish in 60 minutes.  The chair is noonedeadpunk. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:00:24 <opendevmeet> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
15:00:24 <opendevmeet> The meeting name has been set to 'openstack_ansible_meeting'
15:00:29 <noonedeadpunk> #topic roll call
15:00:30 <noonedeadpunk> o/
15:00:40 <damiandabrowski[m]> hey!
15:01:14 <mgariepy> half there o/
15:02:12 <jrosser> o/ hello
15:05:15 <noonedeadpunk> #topic office hours
15:07:37 <noonedeadpunk> I think we're super close to merging repo, centos and ubuntu
15:08:07 <noonedeadpunk> I suggest creating milestone release after that
15:08:23 <noonedeadpunk> and would be awesome to make it this week actually... maybe next...
15:09:07 <jrosser> repo server gluster stuff just has a few cleanup patches to merge which i think are now ok
15:09:29 <jrosser> centos is very close, just the pam_env thing to verify and merge
15:09:51 <damiandabrowski[m]> jrosser: have You managed to solve this issue with mounting cephfs after container restart or You need some help?
15:10:08 <jrosser> hmm?
15:10:28 <jrosser> i've not been looking at cephfs
15:10:42 <damiandabrowski[m]> glusterfs* sorry ;)
15:10:52 <jrosser> oooh - do we still have trouble there
15:11:03 <jrosser> oh it was a race condition with systemd wasnt it
15:11:14 <damiandabrowski[m]> yeah, exactly
15:11:17 <jrosser> tbh i have not spent any time on that
15:11:34 <jrosser> also i don't really have any great idea after reading the systemd bug report
15:13:08 <damiandabrowski[m]> i think there might be some workaround with putting systemd-mount behind systemd service
15:13:22 <damiandabrowski[m]> I'll try to spend some time on this during this week but can't promise anything :/
15:15:40 <noonedeadpunk> Ok, next to that I've started creating presentation for project update
15:16:02 <noonedeadpunk> I think deadline for it is end of the week for it to be shown/promoted during summit
15:16:21 <jrosser> something is also wrong with repo server upgrades here https://zuul.opendev.org/t/openstack/build/7e33eaf8cfb5445c92bcbe38647b792d/log/job-output.txt#19556-19558
15:16:30 <jrosser> i've not seen that before
15:17:13 <noonedeadpunk> unarchive issue? interesting...
15:17:21 <noonedeadpunk> Here's the link for presentation https://docs.google.com/presentation/d/1INXXvAY9MuwASYW4eXLye7YvFaosv5K-7XS7jX6AlpI/edit?usp=sharing
15:17:58 <noonedeadpunk> feedback is welcome. And also if somebody wants to participate in writing down sound or add some slides - let me know:)
15:19:01 <damiandabrowski[m]> good job so far!
15:19:09 <jrosser> i wonder if the unarchive starts before the mount is made completely
15:20:14 <NeilHanlon> slides look great so far (:
15:20:40 <noonedeadpunk> jrosser: that would be weird given it's systemd managing mount
15:21:02 <jrosser> yeah, though i don't know if it's asynchronous
15:21:09 <noonedeadpunk> I'd say ansible module should wait until service/mount started
15:22:21 <noonedeadpunk> But it look like some permissions issue tbh...
15:22:29 <noonedeadpunk> `Cannot change ownership to uid 998, gid 33`
15:23:06 <jrosser> oh now thats interesting actually
15:23:51 <jrosser> i don't think that we ensure that the repo / nginx uid and gid are consistent between the repo servers
15:23:58 <jrosser> that could go wrong
15:24:59 <jrosser> though in this upgrade case it should be only one of then makes the archive and unpacks it into the glusterfs
15:25:21 <jrosser> but i had been thinking we need to do something about the uid/gid anyway
15:25:35 <noonedeadpunk> oh yes, actually that's good point
15:25:59 <noonedeadpunk> but are you sure it's about same server?
15:26:17 <noonedeadpunk> as it's infra job, which means there're 3 of them?
15:26:45 <jrosser> thats right
15:26:52 <noonedeadpunk> Maybe we archived on one, and unpackking on another one? But unlikely...
15:27:20 <jrosser> hah well look at the failing tasks - its on two different ones
15:28:19 <noonedeadpunk> we actually need to run_once this https://opendev.org/openstack/openstack-ansible-repo_server/src/branch/master/tasks/repo_pre_install.yml#L80-L88 as well :)
15:28:48 <noonedeadpunk> But good that we didn't
15:28:55 <jrosser> something very odd is happening there
15:29:05 <jrosser> it's running a bunch of tasks against only two of the 3
15:29:29 <noonedeadpunk> I think we have serial: 1, 100%
15:29:44 <noonedeadpunk> so "as designed"?
15:29:57 <noonedeadpunk> https://opendev.org/openstack/openstack-ansible/src/branch/master/playbooks/repo-install.yml#L25
15:30:29 <jrosser> aah right yes
15:31:37 <jrosser> so we need is_first_play_host type of thing like the other roles
15:31:41 <jrosser> i can look at that
15:32:46 <jrosser> it could be at the moment we are lucky that uid/gid are assigned with the same algorithm in 3 identical containers
15:33:05 <jrosser> thats not going to be the case for long lived deployments or cross-OS in the future
15:34:54 <noonedeadpunk> yes, totally
15:35:25 <noonedeadpunk> I haven't actually looked much in Octavia PKI since I pushed path. I hope to get to it tomorrow
15:35:38 <noonedeadpunk> As it;s smth we need to land sooner better
15:39:38 <jrosser> we could look at bugs? i think a bunch have been addressed
15:41:21 <noonedeadpunk> #topic bug triage
15:41:40 <noonedeadpunk> yeah. I can't recall anyting new, but we closed a lot lately
15:42:33 <noonedeadpunk> #link https://bugs.launchpad.net/openstack-ansible/+bug/1973762
15:42:44 <noonedeadpunk> jrosser: I think you was working on this one?
15:43:29 <jrosser> i didnt make a patch, but basically we template out a file that is already provided by the distro
15:43:44 <jrosser> then i think our changes to make it ext4 get set back to btrfs
15:44:02 <noonedeadpunk> uh
15:44:25 <jrosser> i thought we could make a systemd drop-in to override the fs type= which would then be persistent
15:44:31 <jrosser> but i didnt try it yet
15:44:38 <noonedeadpunk> So we basically should be using systemd overrides for that?
15:45:15 <jrosser> theres some vauge comment in the original file about it being a temporary placeholder until some later version of systemd
15:45:20 <noonedeadpunk> I think https://opendev.org/openstack/openstack-ansible-lxc_hosts/commit/337ddf87802c7597b7c51a071eb9829a4b87defa is related
15:45:29 <jrosser> so it might be that we need to template it out, and also to make the drop in as well
15:45:44 <noonedeadpunk> But I'm not sure it makes things better or worse
15:46:13 <jrosser> well, i see this on my Xena deploys today
15:46:59 <noonedeadpunk> Commit is for master only. And I think it makes things even worse
15:47:25 <jrosser> do they both make the same unit though?
15:47:28 <jrosser> how is it worse
15:47:49 <noonedeadpunk> It creates another mount? for same endpoints
15:48:21 <noonedeadpunk> And kind of not ensuring that existing one is stopped/disabled
15:49:17 <noonedeadpunk> I can look at that
15:49:39 <noonedeadpunk> And smth backportable should be made as well
15:49:45 <noonedeadpunk> for stable branches
15:49:54 <jrosser> i have an AIO here and i think that use of systemd_service overwrites the original unit
15:50:01 <jrosser> but iirc the old code does that too
15:50:08 <jrosser> anyway
15:50:58 <noonedeadpunk> Huh, I thought we're placing our units only under /etc/system
15:51:21 <noonedeadpunk> oh, well, if they have same names at the end - it's just an override then
15:51:55 <noonedeadpunk> https://opendev.org/openstack/ansible-role-systemd_mount/src/branch/master/tasks/systemd_mounts.yml#L56
15:52:30 <jrosser> right, in my AIO i have /etc/systemd/system/var-lib-machines.mount
15:52:39 <jrosser> and its our one with ext4
15:54:08 <noonedeadpunk> Then I guess it should be fine :)
15:54:24 <noonedeadpunk> the only part left - backport smth to stable branches
15:55:16 <noonedeadpunk> I'm thinking to just put simple override of fs type to /etc/systemd/system and be done with it
15:55:40 <jrosser> we should look back though eavesdrop to what spatel shared
15:57:51 <spatel> Reading thread
15:58:06 <jrosser> this is what we tread on https://packages.ubuntu.com/search?suite=focal&arch=any&mode=exactfilename&searchon=contents&keywords=var-lib-machines.mount
15:58:49 <spatel> Got it :)
16:00:02 <noonedeadpunk> I think it's quite clear what the issue is...
16:00:21 <noonedeadpunk> #endmeeting