15:01:11 <noonedeadpunk> #startmeeting openstack_ansible_meeting
15:01:11 <opendevmeet> Meeting started Tue Sep  9 15:01:11 2025 UTC and is due to finish in 60 minutes.  The chair is noonedeadpunk. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:01:11 <opendevmeet> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
15:01:11 <opendevmeet> The meeting name has been set to 'openstack_ansible_meeting'
15:01:20 <noonedeadpunk> #topic rollcall
15:01:23 <noonedeadpunk> o/
15:01:30 <damiandabrowski> hi!
15:01:37 * fungi has something for office hours
15:04:56 <noonedeadpunk> #topic office hours
15:05:03 <noonedeadpunk> fungi: floor is yours :)
15:06:02 <fungi> Bridging the gap between community and contributing orgs: survey and metric analyses
15:06:11 <fungi> i'll try to be quick, but there's a lot we dug into... for some background on openstack-wide metrics analysis see ildikov's most recent ml post from a couple of months back:
15:06:19 <fungi> #link https://lists.openstack.org/archives/list/openstack-discuss@lists.openstack.org/message/NTBNI7YIDCWBR6BTPEKVZIODWTVUIOXO/ BtG metrics analysis
15:06:27 <fungi> also could be worthwhile to revisit her previous post in that thread going over the contributor and maintainer survey results (and anyone who hasn't filled those out for epoxy, please see if you can find a few minutes to do that!)
15:06:35 <fungi> as a follow-up activity, we've been doing some team-specific analyses, focusing on about a dozen teams that had multiple survey responses
15:06:42 <fungi> we're early in the process of analyzing these stats with a focus on improving the experience for maintainers and contributors, so for now this is probably a lot of stuff you already know, or at least confirming what you expected
15:06:55 <fungi> a big part of this is establishing a baseline so that we can better gauge whether future attempts at improving have any observable impact, but we aren't at the point where we have much in the way of guidance or recommendations yet
15:07:04 <fungi> we have plans to continue with this sort of surveying and metrics analysis over coming release cycles and have already drafted some updates for the flamingo versions; as for the initial results...
15:07:13 <fungi> the contributor survey had 1 response for openstack ansible and the respondent self-identified as a new contributor who had contributed to at least one other open source project
15:07:22 <fungi> feedback was polarized (either 1 or 4 out of 5) ranking actionable feedback from reviewers and usefulness of automated testing high, but low scores on timely reviews and basically all aspects of the contributor documentation
15:07:30 <fungi> the one challenge they listed for contributing was a lack of familiarity with gerrit
15:07:41 <fungi> the maintainer survey also had only 1 response so far, with more varied scoring (between 1 and 3 out of 5) the highest being for actionable feedback from reviewers and contributor documentation quality, lowest on timely reviewing again
15:07:51 <fungi> the only contributing challenge reported by that maintainer was with getting review attention, and in reviewing changes for others they struggled with a lack of familiarity in some parts of the project
15:07:59 <fungi> looking at metrics we gathered from gerrit for the past 5 development cycles, review activity has not been nearly as bad as the survey responses suggest, which i would chalk up to selection bias and negative experience outweighing positive
15:08:09 <fungi> there was a fairly significant drop in the number of active maintainers and reviewers as well as change volume during the epoxy cycle compared to prior releases, so i don't know whether that might also have anything to do with it
15:08:17 <fungi> interestingly, the median time to review nearly doubled from dalmatian to epoxy (bobcat was worse though) while the average (mean) time to review stayed roughly the same
15:08:29 <fungi> on the other hand, the time to merge changes was about the same between the two cycles so the drop in maintainers and reviewers doesn't seem to have impacted that, probably owing to a similar drop in changes proposed
15:09:23 <fungi> while it's hard to draw conclusions from such a small number of responses, some additional focus on revamping or updating the contributor documents, maybe leaning a bit omre into referring out to the general openstack contributor workflow docs might help
15:09:41 <fungi> though hard to know if that was just a one-off impression in that survey response
15:09:57 <noonedeadpunk> there's really a lot to unpack here
15:10:21 <fungi> and as is probably obvious, trying to claw back some of the maintainer/reviewer activity, though i don't have great suggestions there yet
15:10:25 <fungi> sorry, i know that's a pretty big info dump (i tried to pare it down as much as possible), and i'm happy to answer questions or take feedback either here in the meeting or any time after in #openstack-manila too
15:10:41 <fungi> er, in #openstack-ansible i mean
15:11:54 <noonedeadpunk> So lately we indeed have a decreased activity from core team regarding reviews. And my inner feeling that merge time should have increased at least this cycle in an observable way
15:12:35 <fungi> keep in mind that this was observing the epoxy cycle activity, after the ptg we'll be doing flamingo analyses and surveys
15:12:48 <noonedeadpunk> However when dealing with other projects in OpenStack, I'd estimate review time as above average on the boards still...
15:13:05 <fungi> yes, it looked that way from the stats
15:13:55 <fungi> slow review response has been just about universal in the survey responses for all teams
15:14:05 <noonedeadpunk> and yes, this cycle we do have a huge focus on improving and re-working our documentation, which was historically not up to date
15:14:09 <fungi> even though the stats for most teams suggest it's not all that bad
15:14:35 <noonedeadpunk> so missing or absent documentation is being recognized as the problem.
15:15:08 <noonedeadpunk> which I guess is imporant, that we also see our own issues and trying to address them
15:15:25 <fungi> the psychology of it, i think, is that if you have 10 changes that merge quickly and 1 change that takes ages to get reviewer interest, you'll remember the one bad experience while forgetting the 10 good ones
15:16:21 <noonedeadpunk> well. it could be as well, that you might have an expectation to get your bugfix merged and released by end of the week
15:16:42 <fungi> the challenge with this initial round is that looking at these surveys as a stand-alone snapshot in time isn't terribly useful. comparing responses from one cycle to the next will be a lot more indicative of whether things are improving
15:16:49 <noonedeadpunk> while it would take a week-two to  just to land on master and then some time for backports
15:17:32 <noonedeadpunk> does this survey also include contributions in form of bug reports?
15:18:23 <noonedeadpunk> as how projects are dealing and some kind of stats for bug fixing would be really also interesting
15:19:06 <fungi> currently the surveys have focused mostly on people proposing changes for review and maintainers reviewing proposed changes, since this effort came out of the tc's request for getting more openinfra member orgs to get involved in development and/or asking why they aren't
15:19:43 <noonedeadpunk> which is fair. gotcha.
15:19:57 <fungi> also the current rate of survey responses makes it hard to get any more granular
15:20:36 <fungi> as i said, openstack ansible had a total of two survey responses (one for the contributor survey, one for the maintainer survey)
15:20:47 <noonedeadpunk> I was just thinking/having an idea of transitioning people reporting bugs to contributors, and why does this happen so rarely
15:21:00 <noonedeadpunk> for instance - because of missing knowledge, time, policies, etc
15:21:13 <noonedeadpunk> just to spreatch survey audience a bit
15:21:17 <fungi> so we're mostly trying to contrast and compare the very few survey responses with available contribution and review metrics
15:21:18 <noonedeadpunk> *streatch
15:21:40 <mgariepy> on issue with patches  (from what i saw on other opensource contribution) is that quite often trying to ping the correct group/person to get stuff moving it really helpful.
15:22:37 <noonedeadpunk> we probably indeed need to update our contributing guidelines and info on how to contact the team with Matrix bridge setup
15:22:45 <fungi> yeah, some of the member orgs we heard from during early outreach were trying to fix bugs that they reported, but were having trouble getting their proposed fixes merged for various reasons (most of which boiled down to miscommunication both on their part and on the part of reviewers who engaged)
15:22:54 <fungi> that wasn't for openstack ansible specifically though
15:24:07 <fungi> mgariepy: definitely, one we were prompted to look at for nova sat unreviewed for a while until the proposer asked for help in #openstack-nova... in the middle of the night on december 26th
15:24:34 <noonedeadpunk> I can recall that one....
15:24:34 <fungi> and then when they (unsurprisingly for most of us i'm sure) got no response there, they gave up
15:25:16 <noonedeadpunk> I wonder if adding something from review stats, like average review time, to contributor docs may manage expectations
15:25:20 <fungi> so knowing who to reach out to and how, but also sometimes when
15:25:24 <noonedeadpunk> and improve overall feeling of time
15:25:58 <noonedeadpunk> as more we talk, more it feels that it's unmatched expectations which impact the most
15:26:14 <fungi> yeah, setting (realistic) expectations in documentation is great and something we're trying to encourage
15:26:36 <fungi> and i agree, miscommunication around expectations is at the root of most of the specific cases we looked into
15:27:12 <fungi> especially around prioritization of reviewing and development areas in projects
15:27:54 <noonedeadpunk> I think also relatively recently on a cinder meeting there were bunch of driver maintainers who also were not really aware of review process, all asking at the beginning of the meeting for reviews, pretty much making it messy and hard to follow
15:28:22 <noonedeadpunk> and then appeared there's some etherpad, which took me l,ike 10mins at least to find among prior meeting logs...
15:28:22 <fungi> from a contributor perspective, it's things like how to know what the team is prioritizing, how (if possible) to get your work prioritized and/or participate in the prioritization decisions
15:28:24 <noonedeadpunk> so eh...
15:28:52 <noonedeadpunk> we have a long road as a community to be explicit about processes...
15:29:06 <fungi> absolutely, this is nothing we're going to fix overnight
15:29:50 <fungi> i'm hoping as a commuinity we can share ideas for what's already working well in some places, make incremental improvements over time, et cetera
15:29:50 <noonedeadpunk> but that is really interesting info, and I think we can have couple of actionable items out of it
15:30:31 <noonedeadpunk> I assume these averages from ML per-project specifically could be taken out of Bitergia relatively easily?
15:30:46 <fungi> as for survey updates, we're adding questions about prioritization to the flamingo round too
15:31:05 <fungi> noonedeadpunk: yes exactly, we pulled them straight from bitergia
15:31:52 <fungi> though tried to avoid getting into comparing teams to each other since their challenges differ and any comparison would be unfair and possibly divisive
15:32:35 <noonedeadpunk> ++
15:33:40 <noonedeadpunk> eventually we have another issue about contributing docs
15:33:41 <fungi> though in cases where teams had particularly positive results we've tried to dig into potential reasons, to see if there are any ideas or workflows that can be transplanted more widely
15:33:45 <noonedeadpunk> which is that we have 2 of them
15:33:50 <noonedeadpunk> #link https://docs.openstack.org/openstack-ansible/latest/contributors/contributing.html
15:34:01 <noonedeadpunk> this one ^was added as part of community goal
15:34:16 <noonedeadpunk> #link https://docs.openstack.org/openstack-ansible/latest/contributors/contribute.html
15:34:30 <noonedeadpunk> and this one is historical, containing some extra info (potentially)
15:34:39 <fungi> probably worth combining them and redirecting one to the other
15:35:42 <fungi> anyway, that's all i had for now, didn't want to monopolize the entire meeting
15:36:01 <noonedeadpunk> I think why I didn't do that, as the one from communty goal was having kinda template. And I treated it as a "exclusive" to bring alignment between projects
15:37:01 <noonedeadpunk> but I think I need to revise this and indeed merge things together....
15:37:02 <fungi> been a while since i read through the goal text, but i thought it was more about making sure some specific details were consistently included by every project, not necessarily limiting what other information could be included there
15:37:55 <fungi> though also it could link out to other pages as well
15:38:39 <noonedeadpunk> yeah, will try to figure this out, as it's obviously a thing which brings some confusion at least
15:38:53 <noonedeadpunk> ok, cool
15:38:55 <NeilHanlon> heya, sorry i am very late :)
15:38:59 <NeilHanlon> at the airport right now
15:39:19 <fungi> we left you a ton of scrollback to entertain you on your flight
15:39:29 <noonedeadpunk> heh, no worries :) that's a good excuse NeilHanlon :)
15:39:33 <noonedeadpunk> ++
15:39:50 <fungi> hopefully lots of food for thought, at least for now
15:39:53 <NeilHanlon> :) was sitting to work on my slides and remembered the time
15:40:21 <NeilHanlon> speaking of, does anyone have real-world OpenStack/NVMe performance numbers they're able to share/point at for my reference?
15:40:49 <noonedeadpunk> no, not really. 99% of our footprint is ceph
15:41:43 <fungi> mnaser might? i recall him doing a lot of comparative benchmarking for local nvme storage on compute nodes in vexxhost
15:41:56 <noonedeadpunk> so, epel has promoted python3-lxc for EL10, so now we finally can drop usage of NeilHanlon COPR
15:42:05 <noonedeadpunk> #link https://review.opendev.org/c/openstack/openstack-ansible-lxc_hosts/+/959415
15:42:28 <noonedeadpunk> and with backport to epoxy, I think we should be able to tag 31.1.0 finally
15:42:45 * noonedeadpunk need to check for other waiting backport
15:43:49 <noonedeadpunk> #link https://review.opendev.org/c/openstack/openstack-ansible-os_skyline/+/958970
15:44:00 <noonedeadpunk> is one would be nice to include though
15:44:05 <noonedeadpunk> but I think it should be it
15:44:33 <noonedeadpunk> on the dark side - there has been no progress with systemd-networkd addition to EPEL
15:45:00 <noonedeadpunk> despite there were suggestions for co-maintenance, they were ignored so far
15:45:10 <NeilHanlon> noonedeadpunk: well actually.. sorta looking at the intersection of Ceph and NVME, so if you're using nvme backed OSDs (a la BlueStore) i'm interested :D
15:45:27 <NeilHanlon> i will ping rsc directly about the systemd thing
15:45:35 <NeilHanlon> see if I can un-stick that
15:46:31 <NeilHanlon> also i guess we should have a couple more packages for rocky 10 that mnasiadka requested -- the openvswitch/ovn and ceph squid repos
15:46:39 <NeilHanlon> (from centos SIGs)
15:46:58 <noonedeadpunk> #link https://bugzilla.redhat.com/show_bug.cgi?id=2303892
15:47:30 <noonedeadpunk> well, while openvswitch/ovn are kinda required - I have not tested if centos sig versions will be working or not
15:47:47 <noonedeadpunk> but ceph is known not to work for some time indeed.
15:48:09 <NeilHanlon> yeah, i dunno the whole state either but I just got asked to provide the -release packages
15:48:28 <noonedeadpunk> we do have nvme osds... will try to check if we have any statistically good performance data...
15:49:08 <NeilHanlon> appreciate that if you can share! I have some of my own data but being able to back it up would be nice lol
15:50:51 <mnasiadka> NeilHanlon: any idea when the compose will land with these extras centos-release-* packages?
15:58:23 <NeilHanlon> today or tomorrow i expect
15:58:58 <NeilHanlon> (mnasiadka)
15:59:04 <mnasiadka> thanks
16:00:05 <noonedeadpunk> #endmeeting