18:00:38 #startmeeting tc 18:00:38 Meeting started Tue Nov 12 18:00:38 2024 UTC and is due to finish in 60 minutes. The chair is gouthamr. Information about MeetBot at http://wiki.debian.org/MeetBot. 18:00:38 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 18:00:38 The meeting name has been set to 'tc' 18:00:56 Welcome to the weekly meeting of the OpenStack Technical Committee. A reminder that this meeting is held under the OpenInfra Code of Conduct available at https://openinfra.dev/legal/code-of-conduct. 18:00:58 Today's meeting agenda can be found at https://wiki.openstack.org/wiki/Meetings/TechnicalCommittee 18:01:01 #topic Roll Call 18:01:04 o/ 18:01:11 o/ 18:01:23 o/ 18:01:23 \o 18:01:29 * JayF waves from the gallery 18:01:35 o/ 18:01:38 \o 18:01:45 JayF: it's better here, we have popcorn ;) 18:03:01 :D 18:04:16 courtesy-ping: bauzas, spotz[m] 18:04:29 doh, forgot 18:04:33 \o 18:05:22 alright, lets get started.. 18:05:35 #topic Last Week's AIs 18:05:58 ^ there weren't any.. we do have some carry over from prior weeks 18:06:04 and from the PTG 18:06:31 we'll discuss those alongside the TC tracker topic.. 18:06:55 was anyone working on anything that they'd like to share? 18:07:59 * gouthamr looks at gmann's patch against openstackclient: https://review.opendev.org/c/openstack/python-openstackclient/+/931858 18:08:17 ^ still wip; can move it to the tracker.. 18:08:23 i did not get chance to work on this, was busy on noble fixes 18:08:35 gouthamr: sure 18:08:54 thanks gmann 18:09:22 alright, lets step into this week's items 18:09:27 #topic Support for third party CI systems (clarkb) 18:09:53 hello 18:10:27 hi clarkb; i think this was in reference to this question on the openstack-discuss mailing list 18:10:31 this came up because I found that someone trying to set up third party CI for cinder is basically asking the zuul mailing list for "how do I run cinder third party CI" 18:10:35 #link https://lists.openstack.org/archives/list/openstack-discuss@lists.openstack.org/message/TT4UZCVBQP3RRWKDCDRS5YVJOTC64BZG/ ([cinder][CI] Can't start zuul service) 18:10:58 oh thats a different thread but I believe the same person 18:11:54 anyway its clear people are struggling to meet the requirements of the Cinder project and I think Zuul is actually the wrong venue for this support. If you have questions about specific zuul features, issues, behaviors then by all means reach out to zuul. The problem here is this person is specifically struggling with the requirement set by Cinder and ideally would get Cinder to 18:11:56 help them 18:12:07 #link https://lists.zuul-ci.org/archives/list/zuul-discuss@lists.zuul-ci.org/thread/IMVDWF5MPTWGZR6Q4WSJZJUT7TNPBES2/ 18:12:09 on top of that I know Cinder isn't the only project that requires third party CI for drivers that can't otherwise be tested upstream 18:12:17 #link https://lists.zuul-ci.org/archives/list/zuul-discuss@lists.zuul-ci.org/thread/6GOTTK25VF7CY57JFR2PYIG476MU7BNK/ 18:12:28 they had a couple of threads on zuul-discuss 18:12:47 the ideal state here is probably that cinder and glance and nova and neutron etc would work together to solve these problems once (maybe with some documentation and possibly a couple people who are able to help out as things change over time) 18:14:24 I wanted to call it out here for that reason (its a bigger thing that just Cinder) and also wanted to point out that third party ci isn't necessarily the easiest thing to do and if we're going to require it then maybe we should be helping make it possible 18:14:45 yes; i think we lost a lot of mindshare when the group that maintained this disbanded: https://wiki.openstack.org/wiki/ThirdPartySystems 18:15:11 what exact problem needs to be solved ? 18:15:17 is it a doc thing ? 18:15:24 are we missing some feature ? 18:15:25 and documentation maintained by project teams is out of date: 18:15:35 not offloading third-party ci configuration help requests onto the zuul maintainers or opendev sysadmins, mainly 18:15:37 #link https://docs.openstack.org/neutron/latest/contributor/policies/thirdparty-ci.html (Neutron Third Party CI docs) 18:15:55 bauzas: I think its two things: 1) documentation that shows you how to run a third party ci that meets the requirements of cinder etc and 2) people who pay attention to those threads when they pop up and point people in the right direction 18:16:25 we recently had the experience of some team wanting to add a new 3rd party CI for vmware driver 18:16:25 #link https://docs.openstack.org/cinder/latest/drivers-all-about.html#new-driver-ci-requirements (cinder CI requirements) 18:16:28 and ya addressing ^ takes that assumed responsiblity off of the zuul and opendev maintainers 18:16:44 AFAIK, they had no problems into jumping into the wagon 18:16:46 there aren't many people in cinder to even review/help with bugfixes 18:17:11 Ironic stopped requiring third-party CI a while ago because logistically it was incredibly difficult to keep running. 18:17:45 JayF: thats also worth calling out. This is a self imposed requirement. If projects are not getting sufficient value compared to the cost then dropping the requirement is also reasonable 18:17:47 #link https://docs.openstack.org/manila/latest/contributor/driver_requirements.html#continuous-integration-systems (manila CI requirements) 18:17:52 maybe we should include cinder team here or discuss in cinder channel otherwise we will not be able to what exactly cinder plan is or was in past for 3rd party CI testing 18:17:54 #link https://docs.openstack.org/nova/latest/contributor/code-review.html#third-party-tests (nova third party CI requirements) 18:17:59 as fungi points out though my person agenda here is to stop having people assume I'm on the hook for it 18:17:59 on the opposite, we got success in *requiring* 3rd-party CI for our driver :) 18:18:25 yeah, for nova it is much less but for cinder there are many 18:18:30 clarkb: to be fair; we also benefited from our primary interface point getting much more standard (redfish) 18:18:55 to a great extent, it's concern for the negative experience those driver maintainers are going to have when they go asking zuul and opendev mailing lists for help and get only crickets (or at best get told to go talk to openstack) 18:19:49 if that helps, we have a specific topic in our weekly meeting for discussing with the particular VMware 3rd-party CI 18:19:57 *nova weekly meeting 18:20:17 that gives the opportunity for the 3rd-party CI maintainer to show up with problems to raise 18:20:23 So it sounds like a bad assumption was made that zuul and opendev folks pick up the third party CI 18:20:28 and a reminder for teams to only set up requirements/expectations for driver maintainers if the team is willing to take on responsibility for guiding them 18:20:53 we had a long time ago some webtool that was giving us the state of 3rd-party CI vendors 18:21:00 bauzas: ya it sounds like nova has solved 2) for nova in that way 18:21:16 maybe other projects requiring third party ci could do something similar to create that contact point 18:21:37 ... or give a POC 18:22:23 anyway I don't really have specific solutions I just wanted to call out that we're effectively ignoring a set of people trying to meet our own demands because no one is willing to make those requirements possible when people struggle 18:22:24 maybe project teams can collaborate and form a group of people interested in sharing their recipes 18:22:26 Yeah there should be a contact way. 18:22:39 i.e., restart a "third party CI" SIG 18:23:15 yes, basically that existed once upon a time 18:23:39 SIG? i think this is particular cinder driver 3rd party CI things and we need to ask their help here? 18:23:52 gmann: it isn't just cinder 18:23:56 the current example is cinder 18:24:07 it's potentially common across all teams who have third-party ci testing requirements 18:24:11 yes; i know a lot of vendor driver maintainers have the same problem with manila as well 18:24:19 but this has been an ongoing problem since third party ci was a thing with varying degrees of success/problems depending on how engaged peopel are 18:25:15 mainly, the people who successfully ran third-party ci systems and were willing to take time out of their busy schedules to help other people who were struggling to do the same thing. it seems like those people are no longer around 18:25:33 honestly saying I am little confused here. what is issue? 3rd party CI maintainer need help on general setup of CI ? or some project side requirement testing? 18:25:50 what gmann said 18:25:51 or not having doc for both 18:25:51 gmann: the issue is that no one within openstack responds to anyone asking about third party CI 18:26:00 at the same time many openstack projects require third party ci 18:26:16 clarkb: that's an unfair assumption 18:26:26 that disconnect results in people grasping for help anywhere they think they can find it which has them land on my doorstep and I'm not in a position to support these folks 18:26:36 clarkb: no response you mean on openstack-discuss or zuul ML? 18:26:39 gmann: maybe a good summary is what i said above, teams should only set requirements/expectations for driver maintainers if the team is willing to take on responsibility for guiding them, because the zuul maintainers and opendev sysadmins aren't going to 18:26:41 gmann: either 18:26:52 that ties to how people interact with the community : we don't read minds 18:27:27 fungi: sure, I am saying they can reachout to project team on ML or IRC channel 18:27:54 fwiw I reached out to cinder on irc for them and got nothing 18:27:58 and I'm surprised that the project community members wouldn't be interested in gaining more test coverage on a vendor 18:28:11 but more generally I think that major requirements liek that should be backed up by some sort of support framework 18:28:24 maybe the problem is that some projects don't require 3rd-party CI for their drivers 18:28:27 because I am not sure how many of cinder team member might be seeing zuul ML but reaching out to them on openstack-discuss or IRC is better way and that is what we can tell 3rd party CI maintainer 18:28:27 documentation and clear communication channels for example (though I could see other systems being used instead or in addition) 18:28:42 gmann: there was also a post on openstack-discuss 18:28:44 they are going to zuul and opendev because there isn't a clear path within openstack first 18:29:03 in the past, nova did a bunch of cleanup to remove false assumptions about the level of support we were shipping for variants of code 18:29:33 that is why i mentioned earlier we should include cinder team in this discussion and know what is their plan for 3rd party CI. because its their requirement to keep backend driver if there is CI and we can ask them to help create doc if nothing 18:30:00 maybe cinder should consider the same and raise the bar for defining supported their vendor drivers 18:30:03 sure I just don't want us to think this is a cinder problem 18:30:04 let's invite the cinder team for the next meeting and see who shows up? 18:30:11 gmann: the timeline is that they asked in openstack-discuss, got no answer after waiting for weeks, then decided to see if zuul-discuss could answer their questions about setting up a third-party ci for cinder 18:30:13 its an lopenstack problem that we're currently seeing crop up with cinder specifically 18:30:16 some more context for you folks: cinder and manila teams tried getting people to share their third party CI setup guides in the past.. we had presentations from NetApp, Pure Storage (and some others).. i think we didn't formalize things into a common place.. 18:30:41 fungi: k 18:31:00 we've to remember the stewards for these teams don't run third party CI themselves.. so its all about coaxing busy vendor maintainers to come to PTG/Summit and submit talks.. 18:31:59 but i like the idea that clarkb and fungi are throwing out.. they are seeking project teams to have a guide in the doc and some PoCs to guide people attempting this 18:31:59 i don't personally care whether this specific incident was for a cinder driver, it's about _openstack_ teams taking responsibility for guiding people when they set expectations for their contributions, rather than those expectations creating more work for someone else 18:33:02 I just pinged cinder PTL about it and see if they can see this discussion and explain what exactly lacking or can do something Nova (other project) did (as bauzas mentioned ) 18:33:03 so that can certainly be a goal; i can take an AI about this and chat with jbernard and carloss and see what we can put together 18:33:06 ya I think two straightforward things that can be done are to have documentation that shows you how to bootstrap a system and also to get people/projects/groups requiring this stuff sign up to respond to questions 18:33:31 but I also don't really care what the solutions are as long as we close the feedback loop here and actually interact with peopel struggling to meet the requirements we set for them 18:34:10 gmann: as I said, there is a difference between nova and cinder expectations about drivers support 18:34:13 It may be important to not lose what fungi pointed at: these folks did reach out on the mailing list and got silence. It's not just about responsibility to opendev to not tax them with questions; we also owe it to the vendors to be responsive if we have that requirement. 18:34:14 clarkb: fungi: can you please link the general doc for 3rd party CI setup which project can link in their specific doc/requirement. that can be reused. 18:34:15 right, as jay mentioned, one possible solution is to just reduce or eliminate expectations that the team itself doesn't have bandwidth to support directly 18:34:16 clarkb: ^ +1.. vendors ("third party") are contributors too, and we need them.. 18:34:35 bauzas: yeah but discussing it in meeting is good way to involve/listen their issue 18:34:35 fungi++ exactly 18:34:38 this is why we usually take more care about 3rd party CI for virt drivers that we can't test in upstream CI 18:34:48 gmann: I think that specifically is what is missing 18:34:53 gmann: it needs to be created by openstack 18:35:09 there's this generic guide: 18:35:09 #link https://docs.opendev.org/opendev/system-config/latest/third_party.html 18:35:28 there is a very outdated doc which i'd prefer opendev dropped entirely: 18:35:29 yeah this one ^^ I was searching 18:35:30 #link https://docs.opendev.org/opendev/system-config/latest/third_party.html 18:35:45 thats a "this is how things fundamentally work doc" 18:35:52 not a "this is how you deploy zuul or jenkins to run a third party ci" doc 18:36:06 its like an api doc for opendev ci reporting 18:36:14 k 18:36:21 what people need is the application that talks to the api and instructions on how to configure it for their use case 18:36:33 +1; please don't drop this guide unless its misinformation :) 18:36:50 I am sure project team might not be expert in 'this is how you deploy zuul or jenkins to run a third party ci' 18:37:01 gmann: then they should drop the requirement 18:37:24 drop requirement of testing means drop driver support 18:37:39 the problem is someone needs to be an expert in it 18:37:42 * gouthamr ~~ time check on this topic ~~ 18:37:43 Ironic maintained a separate project for a while to help folks run bare metal third party CI; especially when hardware is involved (as it is, I assume, with Cinder), it's a big commitment from both project teams and vendors to keep things working. 18:37:47 but what are the options? 1) keep code untested 2) drop 3rd party drivers from upstream 18:38:16 gouthamr: ya we don't need to keep going on this. I just wanted to raise awareness and see if we can find a positive path forward. I think we've managed to raise awareness and solutions can come as people mull it over 18:38:24 frickler: an option (3) there is coordinating with the vendors to do spot testing, which is mostly what we do in Ironic now -- ask vendors or people with hardware to test changes and releases. 18:39:17 JayF: ok, but that would require some persistent effort on the team side 18:39:23 okay good stuff, thanks for bringing this topic here clarkb 18:40:19 and for everyone to pitch in their suggestions, i'll see what we can do in terms of follow ups.. we can continue discussing this on this channel post this meeting; or involve folks that weren't able to make it today 18:40:48 i'd _really_ like PTLs/core maintainers to be paying attention to these meetings, or if they're too tl;dr, the summaries we post after.. 18:41:12 not saying that's not happening :) 18:41:38 because many things do happen due to these meetings in between them.. 18:41:49 #topic 2025.2 Election preparation (gouthamr) 18:41:57 also, let's invite/call project team leaders in such discussion and that way we can have some solution 18:42:56 slaweq and ianychoi were our election officials in the past cycle 18:43:14 bauzas expressed interest in being one for the upcoming election 18:43:47 bauzas expressed to be TC liaison or election official or both? 18:43:59 oh; good question 18:44:01 I'm not opiniated :) 18:44:12 whatever people need 18:44:13 because two are different role 18:44:50 I have some good reasons to consider not running as a contendent for the 2025.2 election 18:44:52 any one else interested in taking one of these roles? 18:45:40 but I still have no light about the post-2025.2 election phase in terms of project leadership :) 18:45:41 I can run as election official again 18:46:07 and TC liaison as well if needed 18:46:49 slaweq: its quite early, do you have any plans to contest the elections? 18:47:29 I have plan to step down after this cycle, it will be 3 years for me and I think I should make space for others :) 18:47:34 "Election officials should not run for election themselves. Ideally they should also have no interest in the election outcome (to preserve neutrality) but that is generally harder to achieve." 18:47:45 * gouthamr doesn't know if we formalized this elsewhere 18:47:47 #link https://wiki.openstack.org/wiki/Election_Officiating_Guidelines 18:47:55 bauzas: yeah good point. Even PTL candidate (not just TC candidate) should not be election official 18:48:02 so if needed, I can be election official again 18:48:17 thank you slaweq 18:48:18 but if there are others who wants to do it this time - even better :) 18:48:28 nope, more the merrier, please stick on 18:48:48 and lets get the conversation started with you two on #openstack-election 18:48:51 gmann: that's where the grey area remains. I'd be very happy to leave my current PTL seat but I have no plan B 18:48:53 (yet) 18:49:03 gouthamr: not sure how much wiki page is maintained but this is latest and up to date doc 18:49:05 #link https://governance.openstack.org/election/#election-officials 18:49:17 ^ yes, we don't have this language there, gmann 18:49:23 i can go add it 18:49:45 bauzas: yeah 18:50:43 i'll also email the community asking for volunteers.. ianychoi has done this a bunch, i don't know if he's tired yet :) 18:50:55 that's all i had for $topic.. 18:51:03 anything else about it? 18:52:01 #topic State of Watcher launchpad team and recovery 18:52:05 #link https://lists.openstack.org/archives/list/openstack-discuss@lists.openstack.org/thread/QLHS64DL7QJR725CEKGAVKCOEKXBE36Z/ (Context) 18:52:20 so just an update as you may have seen on this channel earlier.. 18:53:10 i am chatting with David Tardivel, the owner of "watcher-drivers", but, he didn't respond.. i looked at the email associated with his launchpad account, and noticed that it is his work email pertaining to a former employer 18:53:50 yeah, B42 AFAICR 18:53:51 sigh; i think we need people to know if there's a chance, never to use work emails for open source contributions.. 18:54:40 so i reached out to launchpad to see if we can gain control of the team 18:54:53 #link https://answers.launchpad.net/launchpad/+question/819336 (watcher-drivers ownership request) 18:55:28 we probably need to go down the list of bug trackers on launchpad and ensure we don't have individuals owning them to avoid problems like this ... 18:55:34 I can connect to David, if needed 18:55:54 he apparently switched to another employer 18:56:06 but he could still have his launchpad account 18:56:08 bauzas: please do.. LinkedIn tells me he's reading my messages, but, i can imagine the predicament he's in to respond 18:56:15 that will be great 18:56:36 disclaimer : I very barely know him but I met him once :) 18:56:42 so that may help :) 18:57:04 ++ 18:57:09 alright, anyone wants to volunteer to do a launchpad audit, for the sake of the community? :) 18:57:34 heya o/ im late but let me know if i can help out in any way 18:58:05 jbernard: \o/ thanks for joining - this meeting is wrapping up in ~2 mins, lets chat in a bit 18:58:12 gouthamr: sure 18:58:24 #topic Open Discussion 18:58:37 ^ sorry, just a couple of minutes left 18:58:50 but, if you want to note something for the minutes and follow up later, please do 19:00:00 I'ev got to run the opendev team meeting over the next hour but I'm happy to discuss furhter after 19:00:13 thank you all for attending today! 19:00:26 this meeting will return next week 19:00:29 #endmeeting