18:00:38 <gouthamr> #startmeeting tc
18:00:38 <opendevmeet> Meeting started Tue Nov 12 18:00:38 2024 UTC and is due to finish in 60 minutes.  The chair is gouthamr. Information about MeetBot at http://wiki.debian.org/MeetBot.
18:00:38 <opendevmeet> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
18:00:38 <opendevmeet> The meeting name has been set to 'tc'
18:00:56 <gouthamr> Welcome to the weekly meeting of the OpenStack Technical Committee. A reminder that this meeting is held under the OpenInfra Code of Conduct available at https://openinfra.dev/legal/code-of-conduct.
18:00:58 <gouthamr> Today's meeting agenda can be found at https://wiki.openstack.org/wiki/Meetings/TechnicalCommittee
18:01:01 <gouthamr> #topic Roll Call
18:01:04 <gmann> o/
18:01:11 <slaweq> o/
18:01:23 <gtema> o/
18:01:23 <cardoe> \o
18:01:29 * JayF waves from the gallery
18:01:35 <noonedeadpunk> o/
18:01:38 <frickler> \o
18:01:45 <fungi> JayF: it's better here, we have popcorn ;)
18:03:01 <gouthamr> :D
18:04:16 <gouthamr> courtesy-ping: bauzas, spotz[m]
18:04:29 <bauzas> doh, forgot
18:04:33 <bauzas> \o
18:05:22 <gouthamr> alright, lets get started..
18:05:35 <gouthamr> #topic Last Week's AIs
18:05:58 <gouthamr> ^ there weren't any.. we do have some carry over from prior weeks
18:06:04 <gouthamr> and from the PTG
18:06:31 <gouthamr> we'll discuss those alongside the TC tracker topic..
18:06:55 <gouthamr> was anyone working on anything that they'd like to share?
18:07:59 * gouthamr looks at gmann's patch against openstackclient: https://review.opendev.org/c/openstack/python-openstackclient/+/931858
18:08:17 <gouthamr> ^ still wip; can move it to the tracker..
18:08:23 <gmann> i did not get chance to work on this, was busy on noble fixes
18:08:35 <gmann> gouthamr: sure
18:08:54 <gouthamr> thanks gmann
18:09:22 <gouthamr> alright, lets step into this week's items
18:09:27 <gouthamr> #topic Support for third party CI systems (clarkb)
18:09:53 <clarkb> hello
18:10:27 <gouthamr> hi clarkb; i think this was in reference to this question on the openstack-discuss mailing list
18:10:31 <clarkb> this came up because I found that someone trying to set up third party CI for cinder is basically asking the zuul mailing list for "how do I run cinder third party CI"
18:10:35 <gouthamr> #link https://lists.openstack.org/archives/list/openstack-discuss@lists.openstack.org/message/TT4UZCVBQP3RRWKDCDRS5YVJOTC64BZG/ ([cinder][CI] Can't start zuul service)
18:10:58 <clarkb> oh thats a different thread but I believe the same person
18:11:54 <clarkb> anyway its clear people are struggling to meet the requirements of the Cinder project and I think Zuul is actually the wrong venue for this support. If you have questions about specific zuul features, issues, behaviors then by all means reach out to zuul. The problem here is this person is specifically struggling with the requirement set by Cinder and ideally would get Cinder to
18:11:56 <clarkb> help them
18:12:07 <fungi> #link https://lists.zuul-ci.org/archives/list/zuul-discuss@lists.zuul-ci.org/thread/IMVDWF5MPTWGZR6Q4WSJZJUT7TNPBES2/
18:12:09 <clarkb> on top of that I know Cinder isn't the only project that requires third party CI for drivers that can't otherwise be tested upstream
18:12:17 <fungi> #link https://lists.zuul-ci.org/archives/list/zuul-discuss@lists.zuul-ci.org/thread/6GOTTK25VF7CY57JFR2PYIG476MU7BNK/
18:12:28 <fungi> they had a couple of threads on zuul-discuss
18:12:47 <clarkb> the ideal state here is probably that cinder and glance and nova and neutron etc would work together to solve these problems once (maybe with some documentation and possibly a couple people who are able to help out as things change over time)
18:14:24 <clarkb> I wanted to call it out here for that reason (its a bigger thing that just Cinder) and also wanted to point out that third party ci isn't necessarily the easiest thing to do and if we're going to require it then maybe we should be helping make it possible
18:14:45 <gouthamr> yes; i think we lost a lot of mindshare when the group that maintained this disbanded: https://wiki.openstack.org/wiki/ThirdPartySystems
18:15:11 <bauzas> what exact problem needs to be solved ?
18:15:17 <bauzas> is it a doc thing ?
18:15:24 <bauzas> are we missing some feature ?
18:15:25 <gouthamr> and documentation maintained by project teams is out of date:
18:15:35 <fungi> not offloading third-party ci configuration help requests onto the zuul maintainers or opendev sysadmins, mainly
18:15:37 <gouthamr> #link https://docs.openstack.org/neutron/latest/contributor/policies/thirdparty-ci.html (Neutron Third Party CI docs)
18:15:55 <clarkb> bauzas: I think its two things: 1) documentation that shows you how to run a third party ci that meets the requirements of cinder etc and 2) people who pay attention to those threads when they pop up and point people in the right direction
18:16:25 <bauzas> we recently had the experience of some team wanting to add a new 3rd party CI for vmware driver
18:16:25 <gouthamr> #link https://docs.openstack.org/cinder/latest/drivers-all-about.html#new-driver-ci-requirements (cinder CI requirements)
18:16:28 <clarkb> and ya addressing ^ takes that assumed responsiblity off of the zuul and opendev maintainers
18:16:44 <bauzas> AFAIK, they had no problems into jumping into the wagon
18:16:46 <frickler> there aren't many people in cinder to even review/help with bugfixes
18:17:11 <JayF> Ironic stopped requiring third-party CI a while ago because logistically it was incredibly difficult to keep running.
18:17:45 <clarkb> JayF: thats also worth calling out. This is a self imposed requirement. If projects are not getting sufficient value compared to the cost then dropping the requirement is also reasonable
18:17:47 <gouthamr> #link https://docs.openstack.org/manila/latest/contributor/driver_requirements.html#continuous-integration-systems (manila CI requirements)
18:17:52 <gmann> maybe we should include cinder team here or discuss in cinder channel otherwise we will not be able to what exactly cinder plan is or was in past for 3rd party CI testing
18:17:54 <gouthamr> #link https://docs.openstack.org/nova/latest/contributor/code-review.html#third-party-tests (nova third party CI requirements)
18:17:59 <clarkb> as fungi points out though my person agenda here is to stop having people assume I'm on the hook for it
18:17:59 <bauzas> on the opposite, we got success in *requiring* 3rd-party CI for our driver :)
18:18:25 <gmann> yeah, for nova it is much less but for cinder there are many
18:18:30 <JayF> clarkb: to be fair; we also benefited from our primary interface point getting much more standard (redfish)
18:18:55 <fungi> to a great extent, it's concern for the negative experience those driver maintainers are going to have when they go asking zuul and opendev mailing lists for help and get only crickets (or at best get told to go talk to openstack)
18:19:49 <bauzas> if that helps, we have a specific topic in our weekly meeting for discussing with the particular VMware 3rd-party CI
18:19:57 <bauzas> *nova weekly meeting
18:20:17 <bauzas> that gives the opportunity for the 3rd-party CI maintainer to show up with problems to raise
18:20:23 <cardoe> So it sounds like a bad assumption was made that zuul and opendev folks pick up the third party CI
18:20:28 <fungi> and a reminder for teams to only set up requirements/expectations for driver maintainers if the team is willing to take on responsibility for guiding them
18:20:53 <bauzas> we had a long time ago some webtool that was giving us the state of 3rd-party CI vendors
18:21:00 <clarkb> bauzas: ya it sounds like nova has solved 2) for nova in that way
18:21:16 <clarkb> maybe other projects requiring third party ci could do something similar to create that contact point
18:21:37 <bauzas> ... or give a POC
18:22:23 <clarkb> anyway I don't really have specific solutions I just wanted to call out that we're effectively ignoring a set of people trying to meet our own demands because no one is willing to make those requirements possible when people struggle
18:22:24 <gouthamr> maybe project teams can collaborate and form a group of people interested in sharing their recipes
18:22:26 <cardoe> Yeah there should be a contact way.
18:22:39 <gouthamr> i.e., restart a "third party CI" SIG
18:23:15 <fungi> yes, basically that existed once upon a time
18:23:39 <gmann> SIG? i think this is particular cinder driver 3rd party CI things and we need to ask their help here?
18:23:52 <clarkb> gmann: it isn't just cinder
18:23:56 <clarkb> the current example is cinder
18:24:07 <fungi> it's potentially common across all teams who have third-party ci testing requirements
18:24:11 <gouthamr> yes; i know a lot of vendor driver maintainers have the same problem with manila as well
18:24:19 <clarkb> but this has been an ongoing problem since third party ci was a thing with varying degrees of success/problems depending on how engaged peopel are
18:25:15 <fungi> mainly, the people who successfully ran third-party ci systems and were willing to take time out of their busy schedules to help other people who were struggling to do the same thing. it seems like those people are no longer around
18:25:33 <gmann> honestly saying I am little confused here. what is issue? 3rd party CI maintainer need help on general setup of CI ? or some project side requirement  testing?
18:25:50 <bauzas> what gmann said
18:25:51 <gmann> or not having doc for both
18:25:51 <clarkb> gmann: the issue is that no one within openstack responds to anyone asking about third party CI
18:26:00 <clarkb> at the same time many openstack projects require third party ci
18:26:16 <bauzas> clarkb: that's an unfair assumption
18:26:26 <clarkb> that disconnect results in people grasping for help anywhere they think they can find it which has them land on my doorstep and I'm not in a position to support these folks
18:26:36 <gmann> clarkb: no response you mean on openstack-discuss or zuul ML?
18:26:39 <fungi> gmann: maybe a good summary is what i said above, teams should only set requirements/expectations for driver maintainers if the team is willing to take on responsibility for guiding them, because the zuul maintainers and opendev sysadmins aren't going to
18:26:41 <clarkb> gmann: either
18:26:52 <bauzas> that ties to how people interact with the community : we don't read minds
18:27:27 <gmann> fungi: sure, I am saying they can reachout to project team on ML or IRC channel
18:27:54 <clarkb> fwiw I reached out to cinder on irc for them and got nothing
18:27:58 <bauzas> and I'm surprised that the project community members wouldn't be interested in gaining more test coverage on a vendor
18:28:11 <clarkb> but more generally I think that major requirements liek that should be backed up by some sort of support framework
18:28:24 <bauzas> maybe the problem is that some projects don't require 3rd-party CI for their drivers
18:28:27 <gmann> because I am not sure how many of cinder team member might be seeing zuul ML but reaching out to them on openstack-discuss or IRC is better way and that is what we can tell 3rd party CI maintainer
18:28:27 <clarkb> documentation and clear communication channels for example (though I could see other systems being used instead or in addition)
18:28:42 <fungi> gmann: there was also a post on openstack-discuss
18:28:44 <clarkb> they are going to zuul and opendev because there isn't a clear path within openstack first
18:29:03 <bauzas> in the past, nova did a bunch of cleanup to remove false assumptions about the level of support we were shipping for variants of code
18:29:33 <gmann> that is why i mentioned earlier we should include cinder team in this discussion and know what is their plan for 3rd party CI. because its their requirement to keep backend driver if there is CI and we can ask them to help create doc if nothing
18:30:00 <bauzas> maybe cinder should consider the same and raise the bar for defining supported their vendor drivers
18:30:03 <clarkb> sure I just don't want us to think this is a cinder problem
18:30:04 <frickler> let's invite the cinder team for the next meeting and see who shows up?
18:30:11 <fungi> gmann: the timeline is that they asked in openstack-discuss, got no answer after waiting for weeks, then decided to see if zuul-discuss could answer their questions about setting up a third-party ci for cinder
18:30:13 <clarkb> its an lopenstack problem that we're currently seeing crop up with cinder specifically
18:30:16 <gouthamr> some more context for you folks: cinder and manila teams tried getting people to share their third party CI setup guides in the past.. we had presentations from NetApp, Pure Storage (and some others).. i think we didn't formalize things into a common place..
18:30:41 <gmann> fungi: k
18:31:00 <gouthamr> we've to remember the stewards for these teams don't run third party CI themselves.. so its all about coaxing busy vendor maintainers to come to PTG/Summit and submit talks..
18:31:59 <gouthamr> but i like the idea that clarkb and fungi are throwing out.. they are seeking project teams to have a guide in the doc and some PoCs to guide people attempting this
18:31:59 <fungi> i don't personally care whether this specific incident was for a cinder driver, it's about _openstack_ teams taking responsibility for guiding people when they set expectations for their contributions, rather than those expectations creating more work for someone else
18:33:02 <gmann> I just pinged cinder PTL about it and see if they can see this discussion and explain what exactly lacking or can do something Nova (other project) did (as bauzas mentioned )
18:33:03 <gouthamr> so that can certainly be a goal; i can take an AI about this and chat with jbernard and carloss and see what we can put together
18:33:06 <clarkb> ya I think two straightforward things that can be done are to have documentation that shows you how to bootstrap a system and also to get people/projects/groups requiring this stuff sign up to respond to questions
18:33:31 <clarkb> but I also don't really care what the solutions are as long as we close the feedback loop here and actually interact with peopel struggling to meet the requirements we set for them
18:34:10 <bauzas> gmann: as I said, there is a difference between nova and cinder expectations about drivers support
18:34:13 <JayF> It may be important to not lose what fungi pointed at: these folks did reach out on the mailing list and got silence. It's not just about responsibility to opendev to not tax them with questions; we also owe it to the vendors to be responsive if we have that requirement.
18:34:14 <gmann> clarkb: fungi: can you please link the general doc for 3rd party CI setup which project can link in their specific doc/requirement. that can be reused.
18:34:15 <fungi> right, as jay mentioned, one possible solution is to just reduce or eliminate expectations that the team itself doesn't have bandwidth to support directly
18:34:16 <gouthamr> clarkb: ^ +1.. vendors ("third party") are contributors too, and we need them..
18:34:35 <gmann> bauzas: yeah but discussing it in meeting is good way to involve/listen their issue
18:34:35 <JayF> fungi++ exactly
18:34:38 <bauzas> this is why we usually take more care about 3rd party CI for virt drivers that we can't test in upstream CI
18:34:48 <clarkb> gmann: I think that specifically is what is missing
18:34:53 <clarkb> gmann: it needs to be created by openstack
18:35:09 <gouthamr> there's this generic guide:
18:35:09 <gouthamr> #link https://docs.opendev.org/opendev/system-config/latest/third_party.html
18:35:28 <fungi> there is a very outdated doc which i'd prefer opendev dropped entirely:
18:35:29 <gmann> yeah this one ^^ I was searching
18:35:30 <fungi> #link https://docs.opendev.org/opendev/system-config/latest/third_party.html
18:35:45 <clarkb> thats a "this is how things fundamentally work doc"
18:35:52 <clarkb> not a "this is how you deploy zuul or jenkins to run a third party ci" doc
18:36:06 <clarkb> its like an api doc for opendev ci reporting
18:36:14 <gmann> k
18:36:21 <clarkb> what people need is the application that talks to the api and instructions on how to configure it for their use case
18:36:33 <gouthamr> +1; please don't drop this guide unless its misinformation :)
18:36:50 <gmann> I am sure project team might not be expert in 'this is how you deploy zuul or jenkins to run a third party ci'
18:37:01 <clarkb> gmann: then they should drop the requirement
18:37:24 <gmann> drop requirement of testing means drop driver support
18:37:39 <clarkb> the problem is someone needs to be an expert in it
18:37:42 * gouthamr ~~ time check on this topic ~~
18:37:43 <JayF> Ironic maintained a separate project for a while to help folks run bare metal third party CI; especially when hardware is involved (as it is, I assume, with Cinder), it's a big commitment from both project teams and vendors to keep things working.
18:37:47 <frickler> but what are the options? 1) keep code untested 2) drop 3rd party drivers from upstream
18:38:16 <clarkb> gouthamr: ya we don't need to keep going on this. I just wanted to raise awareness and see if we can find a positive path forward. I think we've managed to raise awareness and solutions can come as people mull it over
18:38:24 <JayF> frickler: an option (3) there is coordinating with the vendors to do spot testing, which is mostly what we do in Ironic now -- ask vendors or people with hardware to test changes and releases.
18:39:17 <frickler> JayF: ok, but that would require some persistent effort on the team side
18:39:23 <gouthamr> okay good stuff, thanks for bringing this topic here clarkb
18:40:19 <gouthamr> and for everyone to pitch in their suggestions, i'll see what we can do in terms of follow ups.. we can continue discussing this on this channel post this meeting; or involve folks that weren't able to make it today
18:40:48 <gouthamr> i'd _really_ like PTLs/core maintainers to be paying attention to these meetings, or if they're too tl;dr, the summaries we post after..
18:41:12 <gouthamr> not saying that's not happening :)
18:41:38 <gouthamr> because many things do happen due to these meetings in between them..
18:41:49 <gouthamr> #topic 2025.2 Election preparation (gouthamr)
18:41:57 <gmann> also, let's invite/call project team leaders in such discussion and that way we can have some solution
18:42:56 <gouthamr> slaweq and ianychoi were our election officials in the past cycle
18:43:14 <gouthamr> bauzas expressed interest in being one for the upcoming election
18:43:47 <gmann> bauzas expressed to be TC liaison or election official or both?
18:43:59 <gouthamr> oh; good question
18:44:01 <bauzas> I'm not opiniated :)
18:44:12 <bauzas> whatever people need
18:44:13 <gmann> because two are different role
18:44:50 <bauzas> I have some good reasons to consider not running as a contendent for the 2025.2 election
18:44:52 <gouthamr> any one else interested in taking one of these roles?
18:45:40 <bauzas> but I still have no light about the post-2025.2 election phase in terms of project leadership :)
18:45:41 <slaweq> I can run as election official again
18:46:07 <slaweq> and TC liaison as well if needed
18:46:49 <gouthamr> slaweq: its quite early, do you have any plans to contest the elections?
18:47:29 <slaweq> I have plan to step down after this cycle, it will be 3 years for me and I think I should make space for others :)
18:47:34 <gouthamr> "Election officials should not run for election themselves. Ideally they should also have no interest in the election outcome (to preserve neutrality) but that is generally harder to achieve."
18:47:45 * gouthamr doesn't know if we formalized this elsewhere
18:47:47 <gouthamr> #link https://wiki.openstack.org/wiki/Election_Officiating_Guidelines
18:47:55 <gmann> bauzas: yeah good point. Even PTL candidate (not just TC candidate) should not be election official
18:48:02 <slaweq> so if needed, I can be election official again
18:48:17 <gouthamr> thank you slaweq
18:48:18 <slaweq> but if there are others who wants to do it this time - even better :)
18:48:28 <gouthamr> nope, more the merrier, please stick on
18:48:48 <gouthamr> and lets get the conversation started with you two on #openstack-election
18:48:51 <bauzas> gmann: that's where the grey area remains. I'd be very happy to leave my current PTL seat but I have no plan B
18:48:53 <bauzas> (yet)
18:49:03 <gmann> gouthamr: not sure how much wiki page is maintained but this is latest and up to date doc
18:49:05 <gmann> #link https://governance.openstack.org/election/#election-officials
18:49:17 <gouthamr> ^ yes, we don't have this language there, gmann
18:49:23 <gouthamr> i can go add it
18:49:45 <gmann> bauzas: yeah
18:50:43 <gouthamr> i'll also email the community asking for volunteers.. ianychoi has done this a bunch, i don't know if he's tired yet :)
18:50:55 <gouthamr> that's all i had for $topic..
18:51:03 <gouthamr> anything else about it?
18:52:01 <gouthamr> #topic State of Watcher launchpad team and recovery
18:52:05 <gouthamr> #link https://lists.openstack.org/archives/list/openstack-discuss@lists.openstack.org/thread/QLHS64DL7QJR725CEKGAVKCOEKXBE36Z/ (Context)
18:52:20 <gouthamr> so just an update as you may have seen on this channel earlier..
18:53:10 <gouthamr> i am chatting with David Tardivel, the owner of "watcher-drivers", but, he didn't respond.. i looked at the email associated with his launchpad account, and noticed that it is his work email pertaining to a former employer
18:53:50 <bauzas> yeah, B42 AFAICR
18:53:51 <gouthamr> sigh; i think we need people to know if there's a chance, never to use work emails for open source contributions..
18:54:40 <gouthamr> so i reached out to launchpad to see if we can gain control of the team
18:54:53 <gouthamr> #link https://answers.launchpad.net/launchpad/+question/819336 (watcher-drivers ownership request)
18:55:28 <gouthamr> we probably need to go down the list of bug trackers on launchpad and ensure we don't have individuals owning them to avoid problems like this ...
18:55:34 <bauzas> I can connect to David, if needed
18:55:54 <bauzas> he apparently switched to another employer
18:56:06 <bauzas> but he could still have his launchpad account
18:56:08 <gouthamr> bauzas: please do.. LinkedIn tells me he's reading my messages, but, i can imagine the predicament he's in to respond
18:56:15 <gmann> that will be great
18:56:36 <bauzas> disclaimer : I very barely know him but I met him once :)
18:56:42 <bauzas> so that may help :)
18:57:04 <gmann> ++
18:57:09 <gouthamr> alright, anyone wants to volunteer to do a launchpad audit, for the sake of the community? :)
18:57:34 <jbernard> heya o/ im late but let me know if i can help out in any way
18:58:05 <gouthamr> jbernard: \o/ thanks for joining - this meeting is wrapping up in ~2 mins, lets chat in a bit
18:58:12 <jbernard> gouthamr: sure
18:58:24 <gouthamr> #topic Open Discussion
18:58:37 <gouthamr> ^ sorry, just a couple of minutes left
18:58:50 <gouthamr> but, if you want to note something for the minutes and follow up later, please do
19:00:00 <clarkb> I'ev got to run the opendev team meeting over the next hour but I'm happy to discuss furhter after
19:00:13 <gouthamr> thank you all for attending today!
19:00:26 <gouthamr> this meeting will return next week
19:00:29 <gouthamr> #endmeeting