| gouthamr | tc-members: a gentle reminder that we’re meeting here in ~45 minutes | 17:15 |
|---|---|---|
| sean-k-mooney | so looking at https://launchpad.net/~watcher-drivers there has not been any movement on changin the owner. did we hwere anything back via linkedin? | 17:25 |
| gouthamr | sean-k-mooney: short answer is no, i was going to bring this up at today's meeting (~23 mins).. i just opened: https://answers.launchpad.net/launchpad/+question/819336 | 17:37 |
| sean-k-mooney | ok no worries | 17:43 |
| gouthamr | #startmeeting tc | 18:00 |
| opendevmeet | Meeting started Tue Nov 12 18:00:38 2024 UTC and is due to finish in 60 minutes. The chair is gouthamr. Information about MeetBot at http://wiki.debian.org/MeetBot. | 18:00 |
| opendevmeet | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 18:00 |
| opendevmeet | The meeting name has been set to 'tc' | 18:00 |
| gouthamr | Welcome to the weekly meeting of the OpenStack Technical Committee. A reminder that this meeting is held under the OpenInfra Code of Conduct available at https://openinfra.dev/legal/code-of-conduct. | 18:00 |
| gouthamr | Today's meeting agenda can be found at https://wiki.openstack.org/wiki/Meetings/TechnicalCommittee | 18:00 |
| gouthamr | #topic Roll Call | 18:01 |
| gmann | o/ | 18:01 |
| slaweq | o/ | 18:01 |
| gtema | o/ | 18:01 |
| cardoe | \o | 18:01 |
| * JayF waves from the gallery | 18:01 | |
| noonedeadpunk | o/ | 18:01 |
| frickler | \o | 18:01 |
| fungi | JayF: it's better here, we have popcorn ;) | 18:01 |
| gouthamr | :D | 18:03 |
| gouthamr | courtesy-ping: bauzas, spotz[m] | 18:04 |
| bauzas | doh, forgot | 18:04 |
| bauzas | \o | 18:04 |
| gouthamr | alright, lets get started.. | 18:05 |
| gouthamr | #topic Last Week's AIs | 18:05 |
| gouthamr | ^ there weren't any.. we do have some carry over from prior weeks | 18:05 |
| gouthamr | and from the PTG | 18:06 |
| gouthamr | we'll discuss those alongside the TC tracker topic.. | 18:06 |
| gouthamr | was anyone working on anything that they'd like to share? | 18:06 |
| * gouthamr looks at gmann's patch against openstackclient: https://review.opendev.org/c/openstack/python-openstackclient/+/931858 | 18:07 | |
| gouthamr | ^ still wip; can move it to the tracker.. | 18:08 |
| gmann | i did not get chance to work on this, was busy on noble fixes | 18:08 |
| gmann | gouthamr: sure | 18:08 |
| gouthamr | thanks gmann | 18:08 |
| gouthamr | alright, lets step into this week's items | 18:09 |
| gouthamr | #topic Support for third party CI systems (clarkb) | 18:09 |
| clarkb | hello | 18:09 |
| gouthamr | hi clarkb; i think this was in reference to this question on the openstack-discuss mailing list | 18:10 |
| clarkb | this came up because I found that someone trying to set up third party CI for cinder is basically asking the zuul mailing list for "how do I run cinder third party CI" | 18:10 |
| gouthamr | #link https://lists.openstack.org/archives/list/openstack-discuss@lists.openstack.org/message/TT4UZCVBQP3RRWKDCDRS5YVJOTC64BZG/ ([cinder][CI] Can't start zuul service) | 18:10 |
| clarkb | oh thats a different thread but I believe the same person | 18:10 |
| clarkb | anyway its clear people are struggling to meet the requirements of the Cinder project and I think Zuul is actually the wrong venue for this support. If you have questions about specific zuul features, issues, behaviors then by all means reach out to zuul. The problem here is this person is specifically struggling with the requirement set by Cinder and ideally would get Cinder to | 18:11 |
| clarkb | help them | 18:11 |
| fungi | #link https://lists.zuul-ci.org/archives/list/zuul-discuss@lists.zuul-ci.org/thread/IMVDWF5MPTWGZR6Q4WSJZJUT7TNPBES2/ | 18:12 |
| clarkb | on top of that I know Cinder isn't the only project that requires third party CI for drivers that can't otherwise be tested upstream | 18:12 |
| fungi | #link https://lists.zuul-ci.org/archives/list/zuul-discuss@lists.zuul-ci.org/thread/6GOTTK25VF7CY57JFR2PYIG476MU7BNK/ | 18:12 |
| fungi | they had a couple of threads on zuul-discuss | 18:12 |
| clarkb | the ideal state here is probably that cinder and glance and nova and neutron etc would work together to solve these problems once (maybe with some documentation and possibly a couple people who are able to help out as things change over time) | 18:12 |
| clarkb | I wanted to call it out here for that reason (its a bigger thing that just Cinder) and also wanted to point out that third party ci isn't necessarily the easiest thing to do and if we're going to require it then maybe we should be helping make it possible | 18:14 |
| gouthamr | yes; i think we lost a lot of mindshare when the group that maintained this disbanded: https://wiki.openstack.org/wiki/ThirdPartySystems | 18:14 |
| bauzas | what exact problem needs to be solved ? | 18:15 |
| bauzas | is it a doc thing ? | 18:15 |
| bauzas | are we missing some feature ? | 18:15 |
| gouthamr | and documentation maintained by project teams is out of date: | 18:15 |
| fungi | not offloading third-party ci configuration help requests onto the zuul maintainers or opendev sysadmins, mainly | 18:15 |
| gouthamr | #link https://docs.openstack.org/neutron/latest/contributor/policies/thirdparty-ci.html (Neutron Third Party CI docs) | 18:15 |
| clarkb | bauzas: I think its two things: 1) documentation that shows you how to run a third party ci that meets the requirements of cinder etc and 2) people who pay attention to those threads when they pop up and point people in the right direction | 18:15 |
| bauzas | we recently had the experience of some team wanting to add a new 3rd party CI for vmware driver | 18:16 |
| gouthamr | #link https://docs.openstack.org/cinder/latest/drivers-all-about.html#new-driver-ci-requirements (cinder CI requirements) | 18:16 |
| clarkb | and ya addressing ^ takes that assumed responsiblity off of the zuul and opendev maintainers | 18:16 |
| bauzas | AFAIK, they had no problems into jumping into the wagon | 18:16 |
| frickler | there aren't many people in cinder to even review/help with bugfixes | 18:16 |
| JayF | Ironic stopped requiring third-party CI a while ago because logistically it was incredibly difficult to keep running. | 18:17 |
| clarkb | JayF: thats also worth calling out. This is a self imposed requirement. If projects are not getting sufficient value compared to the cost then dropping the requirement is also reasonable | 18:17 |
| gouthamr | #link https://docs.openstack.org/manila/latest/contributor/driver_requirements.html#continuous-integration-systems (manila CI requirements) | 18:17 |
| gmann | maybe we should include cinder team here or discuss in cinder channel otherwise we will not be able to what exactly cinder plan is or was in past for 3rd party CI testing | 18:17 |
| gouthamr | #link https://docs.openstack.org/nova/latest/contributor/code-review.html#third-party-tests (nova third party CI requirements) | 18:17 |
| clarkb | as fungi points out though my person agenda here is to stop having people assume I'm on the hook for it | 18:17 |
| bauzas | on the opposite, we got success in *requiring* 3rd-party CI for our driver :) | 18:17 |
| gmann | yeah, for nova it is much less but for cinder there are many | 18:18 |
| JayF | clarkb: to be fair; we also benefited from our primary interface point getting much more standard (redfish) | 18:18 |
| fungi | to a great extent, it's concern for the negative experience those driver maintainers are going to have when they go asking zuul and opendev mailing lists for help and get only crickets (or at best get told to go talk to openstack) | 18:18 |
| bauzas | if that helps, we have a specific topic in our weekly meeting for discussing with the particular VMware 3rd-party CI | 18:19 |
| bauzas | *nova weekly meeting | 18:19 |
| bauzas | that gives the opportunity for the 3rd-party CI maintainer to show up with problems to raise | 18:20 |
| cardoe | So it sounds like a bad assumption was made that zuul and opendev folks pick up the third party CI | 18:20 |
| fungi | and a reminder for teams to only set up requirements/expectations for driver maintainers if the team is willing to take on responsibility for guiding them | 18:20 |
| bauzas | we had a long time ago some webtool that was giving us the state of 3rd-party CI vendors | 18:20 |
| clarkb | bauzas: ya it sounds like nova has solved 2) for nova in that way | 18:21 |
| clarkb | maybe other projects requiring third party ci could do something similar to create that contact point | 18:21 |
| bauzas | ... or give a POC | 18:21 |
| clarkb | anyway I don't really have specific solutions I just wanted to call out that we're effectively ignoring a set of people trying to meet our own demands because no one is willing to make those requirements possible when people struggle | 18:22 |
| gouthamr | maybe project teams can collaborate and form a group of people interested in sharing their recipes | 18:22 |
| cardoe | Yeah there should be a contact way. | 18:22 |
| gouthamr | i.e., restart a "third party CI" SIG | 18:22 |
| fungi | yes, basically that existed once upon a time | 18:23 |
| gmann | SIG? i think this is particular cinder driver 3rd party CI things and we need to ask their help here? | 18:23 |
| clarkb | gmann: it isn't just cinder | 18:23 |
| clarkb | the current example is cinder | 18:23 |
| fungi | it's potentially common across all teams who have third-party ci testing requirements | 18:24 |
| gouthamr | yes; i know a lot of vendor driver maintainers have the same problem with manila as well | 18:24 |
| clarkb | but this has been an ongoing problem since third party ci was a thing with varying degrees of success/problems depending on how engaged peopel are | 18:24 |
| fungi | mainly, the people who successfully ran third-party ci systems and were willing to take time out of their busy schedules to help other people who were struggling to do the same thing. it seems like those people are no longer around | 18:25 |
| gmann | honestly saying I am little confused here. what is issue? 3rd party CI maintainer need help on general setup of CI ? or some project side requirement testing? | 18:25 |
| bauzas | what gmann said | 18:25 |
| gmann | or not having doc for both | 18:25 |
| clarkb | gmann: the issue is that no one within openstack responds to anyone asking about third party CI | 18:25 |
| clarkb | at the same time many openstack projects require third party ci | 18:26 |
| bauzas | clarkb: that's an unfair assumption | 18:26 |
| clarkb | that disconnect results in people grasping for help anywhere they think they can find it which has them land on my doorstep and I'm not in a position to support these folks | 18:26 |
| gmann | clarkb: no response you mean on openstack-discuss or zuul ML? | 18:26 |
| fungi | gmann: maybe a good summary is what i said above, teams should only set requirements/expectations for driver maintainers if the team is willing to take on responsibility for guiding them, because the zuul maintainers and opendev sysadmins aren't going to | 18:26 |
| clarkb | gmann: either | 18:26 |
| bauzas | that ties to how people interact with the community : we don't read minds | 18:26 |
| gmann | fungi: sure, I am saying they can reachout to project team on ML or IRC channel | 18:27 |
| clarkb | fwiw I reached out to cinder on irc for them and got nothing | 18:27 |
| bauzas | and I'm surprised that the project community members wouldn't be interested in gaining more test coverage on a vendor | 18:27 |
| clarkb | but more generally I think that major requirements liek that should be backed up by some sort of support framework | 18:28 |
| bauzas | maybe the problem is that some projects don't require 3rd-party CI for their drivers | 18:28 |
| gmann | because I am not sure how many of cinder team member might be seeing zuul ML but reaching out to them on openstack-discuss or IRC is better way and that is what we can tell 3rd party CI maintainer | 18:28 |
| clarkb | documentation and clear communication channels for example (though I could see other systems being used instead or in addition) | 18:28 |
| fungi | gmann: there was also a post on openstack-discuss | 18:28 |
| clarkb | they are going to zuul and opendev because there isn't a clear path within openstack first | 18:28 |
| bauzas | in the past, nova did a bunch of cleanup to remove false assumptions about the level of support we were shipping for variants of code | 18:29 |
| gmann | that is why i mentioned earlier we should include cinder team in this discussion and know what is their plan for 3rd party CI. because its their requirement to keep backend driver if there is CI and we can ask them to help create doc if nothing | 18:29 |
| bauzas | maybe cinder should consider the same and raise the bar for defining supported their vendor drivers | 18:30 |
| clarkb | sure I just don't want us to think this is a cinder problem | 18:30 |
| frickler | let's invite the cinder team for the next meeting and see who shows up? | 18:30 |
| fungi | gmann: the timeline is that they asked in openstack-discuss, got no answer after waiting for weeks, then decided to see if zuul-discuss could answer their questions about setting up a third-party ci for cinder | 18:30 |
| clarkb | its an lopenstack problem that we're currently seeing crop up with cinder specifically | 18:30 |
| gouthamr | some more context for you folks: cinder and manila teams tried getting people to share their third party CI setup guides in the past.. we had presentations from NetApp, Pure Storage (and some others).. i think we didn't formalize things into a common place.. | 18:30 |
| gmann | fungi: k | 18:30 |
| gouthamr | we've to remember the stewards for these teams don't run third party CI themselves.. so its all about coaxing busy vendor maintainers to come to PTG/Summit and submit talks.. | 18:31 |
| gouthamr | but i like the idea that clarkb and fungi are throwing out.. they are seeking project teams to have a guide in the doc and some PoCs to guide people attempting this | 18:31 |
| fungi | i don't personally care whether this specific incident was for a cinder driver, it's about _openstack_ teams taking responsibility for guiding people when they set expectations for their contributions, rather than those expectations creating more work for someone else | 18:31 |
| gmann | I just pinged cinder PTL about it and see if they can see this discussion and explain what exactly lacking or can do something Nova (other project) did (as bauzas mentioned ) | 18:33 |
| gouthamr | so that can certainly be a goal; i can take an AI about this and chat with jbernard and carloss and see what we can put together | 18:33 |
| clarkb | ya I think two straightforward things that can be done are to have documentation that shows you how to bootstrap a system and also to get people/projects/groups requiring this stuff sign up to respond to questions | 18:33 |
| clarkb | but I also don't really care what the solutions are as long as we close the feedback loop here and actually interact with peopel struggling to meet the requirements we set for them | 18:33 |
| bauzas | gmann: as I said, there is a difference between nova and cinder expectations about drivers support | 18:34 |
| JayF | It may be important to not lose what fungi pointed at: these folks did reach out on the mailing list and got silence. It's not just about responsibility to opendev to not tax them with questions; we also owe it to the vendors to be responsive if we have that requirement. | 18:34 |
| gmann | clarkb: fungi: can you please link the general doc for 3rd party CI setup which project can link in their specific doc/requirement. that can be reused. | 18:34 |
| fungi | right, as jay mentioned, one possible solution is to just reduce or eliminate expectations that the team itself doesn't have bandwidth to support directly | 18:34 |
| gouthamr | clarkb: ^ +1.. vendors ("third party") are contributors too, and we need them.. | 18:34 |
| gmann | bauzas: yeah but discussing it in meeting is good way to involve/listen their issue | 18:34 |
| JayF | fungi++ exactly | 18:34 |
| bauzas | this is why we usually take more care about 3rd party CI for virt drivers that we can't test in upstream CI | 18:34 |
| clarkb | gmann: I think that specifically is what is missing | 18:34 |
| clarkb | gmann: it needs to be created by openstack | 18:34 |
| gouthamr | there's this generic guide: | 18:35 |
| gouthamr | #link https://docs.opendev.org/opendev/system-config/latest/third_party.html | 18:35 |
| fungi | there is a very outdated doc which i'd prefer opendev dropped entirely: | 18:35 |
| gmann | yeah this one ^^ I was searching | 18:35 |
| fungi | #link https://docs.opendev.org/opendev/system-config/latest/third_party.html | 18:35 |
| clarkb | thats a "this is how things fundamentally work doc" | 18:35 |
| clarkb | not a "this is how you deploy zuul or jenkins to run a third party ci" doc | 18:35 |
| clarkb | its like an api doc for opendev ci reporting | 18:36 |
| gmann | k | 18:36 |
| clarkb | what people need is the application that talks to the api and instructions on how to configure it for their use case | 18:36 |
| gouthamr | +1; please don't drop this guide unless its misinformation :) | 18:36 |
| gmann | I am sure project team might not be expert in 'this is how you deploy zuul or jenkins to run a third party ci' | 18:36 |
| clarkb | gmann: then they should drop the requirement | 18:37 |
| gmann | drop requirement of testing means drop driver support | 18:37 |
| clarkb | the problem is someone needs to be an expert in it | 18:37 |
| * gouthamr ~~ time check on this topic ~~ | 18:37 | |
| JayF | Ironic maintained a separate project for a while to help folks run bare metal third party CI; especially when hardware is involved (as it is, I assume, with Cinder), it's a big commitment from both project teams and vendors to keep things working. | 18:37 |
| frickler | but what are the options? 1) keep code untested 2) drop 3rd party drivers from upstream | 18:37 |
| clarkb | gouthamr: ya we don't need to keep going on this. I just wanted to raise awareness and see if we can find a positive path forward. I think we've managed to raise awareness and solutions can come as people mull it over | 18:38 |
| JayF | frickler: an option (3) there is coordinating with the vendors to do spot testing, which is mostly what we do in Ironic now -- ask vendors or people with hardware to test changes and releases. | 18:38 |
| frickler | JayF: ok, but that would require some persistent effort on the team side | 18:39 |
| gouthamr | okay good stuff, thanks for bringing this topic here clarkb | 18:39 |
| gouthamr | and for everyone to pitch in their suggestions, i'll see what we can do in terms of follow ups.. we can continue discussing this on this channel post this meeting; or involve folks that weren't able to make it today | 18:40 |
| gouthamr | i'd _really_ like PTLs/core maintainers to be paying attention to these meetings, or if they're too tl;dr, the summaries we post after.. | 18:40 |
| gouthamr | not saying that's not happening :) | 18:41 |
| gouthamr | because many things do happen due to these meetings in between them.. | 18:41 |
| gouthamr | #topic 2025.2 Election preparation (gouthamr) | 18:41 |
| gmann | also, let's invite/call project team leaders in such discussion and that way we can have some solution | 18:41 |
| gouthamr | slaweq and ianychoi were our election officials in the past cycle | 18:42 |
| gouthamr | bauzas expressed interest in being one for the upcoming election | 18:43 |
| gmann | bauzas expressed to be TC liaison or election official or both? | 18:43 |
| gouthamr | oh; good question | 18:43 |
| bauzas | I'm not opiniated :) | 18:44 |
| bauzas | whatever people need | 18:44 |
| gmann | because two are different role | 18:44 |
| bauzas | I have some good reasons to consider not running as a contendent for the 2025.2 election | 18:44 |
| gouthamr | any one else interested in taking one of these roles? | 18:44 |
| bauzas | but I still have no light about the post-2025.2 election phase in terms of project leadership :) | 18:45 |
| slaweq | I can run as election official again | 18:45 |
| slaweq | and TC liaison as well if needed | 18:46 |
| gouthamr | slaweq: its quite early, do you have any plans to contest the elections? | 18:46 |
| slaweq | I have plan to step down after this cycle, it will be 3 years for me and I think I should make space for others :) | 18:47 |
| gouthamr | "Election officials should not run for election themselves. Ideally they should also have no interest in the election outcome (to preserve neutrality) but that is generally harder to achieve." | 18:47 |
| * gouthamr doesn't know if we formalized this elsewhere | 18:47 | |
| gouthamr | #link https://wiki.openstack.org/wiki/Election_Officiating_Guidelines | 18:47 |
| gmann | bauzas: yeah good point. Even PTL candidate (not just TC candidate) should not be election official | 18:47 |
| slaweq | so if needed, I can be election official again | 18:48 |
| gouthamr | thank you slaweq | 18:48 |
| slaweq | but if there are others who wants to do it this time - even better :) | 18:48 |
| gouthamr | nope, more the merrier, please stick on | 18:48 |
| gouthamr | and lets get the conversation started with you two on #openstack-election | 18:48 |
| bauzas | gmann: that's where the grey area remains. I'd be very happy to leave my current PTL seat but I have no plan B | 18:48 |
| bauzas | (yet) | 18:48 |
| gmann | gouthamr: not sure how much wiki page is maintained but this is latest and up to date doc | 18:49 |
| gmann | #link https://governance.openstack.org/election/#election-officials | 18:49 |
| gouthamr | ^ yes, we don't have this language there, gmann | 18:49 |
| gouthamr | i can go add it | 18:49 |
| gmann | bauzas: yeah | 18:49 |
| gouthamr | i'll also email the community asking for volunteers.. ianychoi has done this a bunch, i don't know if he's tired yet :) | 18:50 |
| gouthamr | that's all i had for $topic.. | 18:50 |
| gouthamr | anything else about it? | 18:51 |
| gouthamr | #topic State of Watcher launchpad team and recovery | 18:52 |
| gouthamr | #link https://lists.openstack.org/archives/list/openstack-discuss@lists.openstack.org/thread/QLHS64DL7QJR725CEKGAVKCOEKXBE36Z/ (Context) | 18:52 |
| gouthamr | so just an update as you may have seen on this channel earlier.. | 18:52 |
| gouthamr | i am chatting with David Tardivel, the owner of "watcher-drivers", but, he didn't respond.. i looked at the email associated with his launchpad account, and noticed that it is his work email pertaining to a former employer | 18:53 |
| bauzas | yeah, B42 AFAICR | 18:53 |
| gouthamr | sigh; i think we need people to know if there's a chance, never to use work emails for open source contributions.. | 18:53 |
| gouthamr | so i reached out to launchpad to see if we can gain control of the team | 18:54 |
| gouthamr | #link https://answers.launchpad.net/launchpad/+question/819336 (watcher-drivers ownership request) | 18:54 |
| gouthamr | we probably need to go down the list of bug trackers on launchpad and ensure we don't have individuals owning them to avoid problems like this ... | 18:55 |
| bauzas | I can connect to David, if needed | 18:55 |
| bauzas | he apparently switched to another employer | 18:55 |
| bauzas | but he could still have his launchpad account | 18:56 |
| gouthamr | bauzas: please do.. LinkedIn tells me he's reading my messages, but, i can imagine the predicament he's in to respond | 18:56 |
| gmann | that will be great | 18:56 |
| bauzas | disclaimer : I very barely know him but I met him once :) | 18:56 |
| bauzas | so that may help :) | 18:56 |
| gmann | ++ | 18:57 |
| gouthamr | alright, anyone wants to volunteer to do a launchpad audit, for the sake of the community? :) | 18:57 |
| jbernard | heya o/ im late but let me know if i can help out in any way | 18:57 |
| gouthamr | jbernard: \o/ thanks for joining - this meeting is wrapping up in ~2 mins, lets chat in a bit | 18:58 |
| jbernard | gouthamr: sure | 18:58 |
| gouthamr | #topic Open Discussion | 18:58 |
| gouthamr | ^ sorry, just a couple of minutes left | 18:58 |
| gouthamr | but, if you want to note something for the minutes and follow up later, please do | 18:58 |
| clarkb | I'ev got to run the opendev team meeting over the next hour but I'm happy to discuss furhter after | 19:00 |
| gouthamr | thank you all for attending today! | 19:00 |
| gouthamr | this meeting will return next week | 19:00 |
| gouthamr | #endmeeting | 19:00 |
| opendevmeet | Meeting ended Tue Nov 12 19:00:29 2024 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 19:00 |
| opendevmeet | Minutes: https://meetings.opendev.org/meetings/tc/2024/tc.2024-11-12-18.00.html | 19:00 |
| opendevmeet | Minutes (text): https://meetings.opendev.org/meetings/tc/2024/tc.2024-11-12-18.00.txt | 19:00 |
| opendevmeet | Log: https://meetings.opendev.org/meetings/tc/2024/tc.2024-11-12-18.00.log.html | 19:00 |
| bauzas | thanks | 19:00 |
| * bauzas goes opening his can | 19:01 | |
| gouthamr | hahaha prost! | 19:01 |
| gouthamr | jbernard: carloss: do catch up with scrollback too, but clarkb pointed out that we have cinder/manila (and other projects) requiring third party CI, but, don't have any documentation on how one would set one up.. are you interested in working together to start a group of people that can help create such a document? and also guide future third party system maintainers when they begin finding issues that the doc can't help with? | 19:03 |
| clarkb | in addition to not having documents to bootstrap people there is also a lack of response/interaction/help when people ask questions | 19:04 |
| sean-k-mooney | nova has required that in the past too | 19:05 |
| clarkb | I think it is also important to remember that opendev and zuul are not experts in how to run third party ci and tests for cinder or manila etc so catching people before they get to us is a good thing | 19:05 |
| sean-k-mooney | there were some docuemnt on how to do that in the past | 19:05 |
| sean-k-mooney | not sure if they are maintained | 19:05 |
| jbernard | sure, it's possible we do have something, i vaguely remember seeing something, ill see if I can track that down (documentation) | 19:05 |
| fungi | sean-k-mooney: i think jay had a blog post about it, in the long-long-ago | 19:06 |
| sean-k-mooney | well there used to be automation to help you set it up and there were some docs around that in the before times | 19:06 |
| JayF | pipes? | 19:06 |
| JayF | not this Jay :) | 19:06 |
| fungi | yeah, sorry, not new-jay ;) | 19:06 |
| sean-k-mooney | https://docs.openstack.org/infra/openstackci/third_party_ci.html | 19:06 |
| sean-k-mooney | ther is that | 19:07 |
| JayF | I overlapped quite a bit with Jay P :P | 19:07 |
| sean-k-mooney | i never actully used that any time i was invovled in setting up a third party ci because i hate using puppet | 19:07 |
| fungi | sean-k-mooney: good find, i forgot that was still hanging out on the internet too. so ancient the cobwebs now have their own cobwebs | 19:07 |
| sean-k-mooney | but i remember using that as a refernce back in 2014 the frist time we tried | 19:08 |
| fungi | it's like a time capsule, last published a decade ago according to the footer | 19:09 |
| sean-k-mooney | yep the high levle steps have not changed really | 19:09 |
| sean-k-mooney | i woudl advise usign zuul v3 and not jenkins | 19:09 |
| clarkb | another thing to keep in mind is that people trying to contribute to openstack aren't going to care if it is cinder or any other project that is failing to provide necessary support. Their takeaway will be that openstack's contribution requirements are too high and that will be that | 19:10 |
| sean-k-mooney | but creating the review account and connectign to the event stream is all the same | 19:10 |
| sean-k-mooney | so the third party ci requirement were really for vendors not indiviugals | 19:10 |
| clarkb | yes and vendors may just decide to stop bothering | 19:11 |
| sean-k-mooney | if intel or dell wanted to enabel feature x that was vendor specific the ask was for them to test it | 19:11 |
| sean-k-mooney | yep | 19:11 |
| clarkb | its a two way street is my point. If we set the bar this high but then can't even reach it ourselves no one will be happy | 19:11 |
| sean-k-mooney | which is fine we jsut remove there code :P | 19:11 |
| fungi | i've seen some indication that there are more individual users of specific hardware who want to maintain a driver these days now that a lot of the vendors have lost interest | 19:11 |
| jbernard | clarkb: is there something specific to cinder that needs attention at the moment? | 19:12 |
| sean-k-mooney | ya, vendor attrision is definally a thing | 19:12 |
| clarkb | jbernard: yes there is someone from fujitsu who has asked all over the place for help running a third party ci system | 19:12 |
| fungi | jbernard: maybe replying to https://lists.openstack.org/archives/list/openstack-discuss@lists.openstack.org/message/TT4UZCVBQP3RRWKDCDRS5YVJOTC64BZG/ | 19:12 |
| clarkb | jbernard: they tried opensatck-discuss and diddn't get a response, they then tried the zuul mailing list and didn't get a response. I called out the zuul thread in the cinder irc room and I didn't get a response nor did the original mailing list email. I'm not sure if anyone reached out privately | 19:13 |
| gmann | yeah that is exactly missing here, replying to the ML with some doc ref if they are new or help as much as we can | 19:13 |
| sean-k-mooney | so i am clonflicted https://github.com/SeanMooney/ci-sean-mooney might help but i dont really have time to discribe this end to end. | 19:14 |
| sean-k-mooney | its actully pretty quick to take the zuul docker compose, make a few tweeks and start locally testing htings | 19:14 |
| sean-k-mooney | moving it to something useful is more work | 19:14 |
| sean-k-mooney | i dont have my nodepool/zuul tenat info in that repo unfortunetly | 19:15 |
| sean-k-mooney | the rdo ci is perhaps the best public refernce fo a third paty ci but its also overly complex | 19:15 |
| sean-k-mooney | since they added an indricion layer through usign dhall for config | 19:16 |
| jbernard | is https://docs.openstack.org/infra/openstackci/third_party_ci.html the best we can point to at the moment, or is there something more up to date? | 19:19 |
| fungi | yes, https://zuul-ci.org/docs/zuul/latest/tutorials/quick-start.html is 90% of what someone would presumably need for the zuul setup itself, adjusted to drop the built-in gerrit container, credentials/pipeline added to talk to opendev's gerrit, nodepool set up for however they're managing their test systems... but then comes adding a devstack-tempest based job | 19:19 |
| fungi | jbernard: the doc you linked is 10 years old and most of what it describes no longer exists | 19:19 |
| sean-k-mooney | https://docs.opendev.org/opendev/system-config/latest/third_party.html is a better stating point | 19:19 |
| jbernard | ok, that's helpful | 19:20 |
| sean-k-mooney | https://docs.opendev.org/opendev/system-config/latest/third_party.html has been updated somehat more recnetly | 19:20 |
| sean-k-mooney | https://zuul-ci.org/docs/zuul/latest/tutorials/quick-start.html can get a working zuul (with a local gerrit) workign pretty quickly | 19:20 |
| fungi | yeah, it's light on details, but more accurate in what it does contain | 19:21 |
| fungi | (the system-config doc i mean) | 19:21 |
| sean-k-mooney | if you follow the steps to create a ci account form the ohter doc you can add an addtion gerrit conenction to the opendev gerrit | 19:21 |
| sean-k-mooney | then you can lcoally start builing out a ci | 19:21 |
| jbernard | clarkb: inori mailed zuul-discuss, how did that go? I'm wondering what state their ci is in. | 19:23 |
| clarkb | jbernard: they didn't get any response on zuul-discuss. zuul-discuss isn't an appropriate venue to figure out how to run cinder third party ci | 19:24 |
| clarkb | we don't know how to run third party ci systems there and a lot of the questions are basically related to "how do I wrestle configs to run devstack/tempest in my system" | 19:25 |
| sean-k-mooney | reusing the upstream tempest jobs is very doable | 19:26 |
| jbernard | ok, ill reach out to inori, all of the errors on the ML are zuul-specific, maybe they've made progress | 19:26 |
| sean-k-mooney | mainly you just need ot create an approate base jobs and nodesets | 19:26 |
| sean-k-mooney | perhaps jobs is a strach but the playbooks are defiently reuabls | 19:27 |
| clarkb | jbernard: the problem is they aren't zuul specific | 19:27 |
| clarkb | jbernard: they are specific to "i'm trying to run cinder third party ci" | 19:27 |
| clarkb | you can't just reuse configs you have to do a lot of stuff to make that work and thats the context that is completely missing for anyone not doing third party ci | 19:27 |
| clarkb | sean-k-mooney: the problem is while doable it requires a lot of third party and tempest/devstack domain specific info on what exactly needs doing | 19:29 |
| sean-k-mooney | yep, you basically have to do this https://github.com/SeanMooney/ci-sean-mooney/blob/main/zuul.d/jobs.yaml | 19:29 |
| sean-k-mooney | i.e. defien you one set of jobs to emulat the upstream ones | 19:29 |
| sean-k-mooney | reduing all the playbooks ectra | 19:29 |
| sean-k-mooney | which mean you have to knwo how those work | 19:29 |
| sean-k-mooney | sap recently set up a third party ci to test vmware | 19:29 |
| sean-k-mooney | as far as im aware they did that with kolla-anisble and jenkins | 19:30 |
| sean-k-mooney | so you dont technially need to use zuul or devetack | 19:30 |
| fungi | the questions posed in https://lists.zuul-ci.org/archives/list/zuul-discuss@lists.zuul-ci.org/thread/IMVDWF5MPTWGZR6Q4WSJZJUT7TNPBES2/ in particular assume a lot of openstack context (recheck comments, reusing jobs defined in openstack repositories and needing to know what other repositories they'll need in their config to make those not error, et cetera) | 19:30 |
| sean-k-mooney | provided you can deploy a system that can take teh patch under review and test it properly | 19:30 |
| sean-k-mooney | my personal experice si reusing entire jobs is not worth it | 19:32 |
| sean-k-mooney | it very hard to do properly. reuisng the same playbooks and 90% of the job definitons you care about is simpler | 19:33 |
| sean-k-mooney | gouthamr: if it make you feel better i rarely but ocationally put the meeting recordign form youtube up while im cooking dinner | 19:35 |
| sean-k-mooney | but i dont often get time to review the summeray or watch them back in general | 19:36 |
| sean-k-mooney | gouthamr: if im aware that somethign interesting was dicussed in the tc meeting form some other souce i will ocationally follow up on the video or if there is an ml post but i generally wotn look for the irc minutes | 19:38 |
| sean-k-mooney | fungi: clarkb one way to solve the third party ci problem woudl be to work with one of the installer to offically supprot all the bits | 19:39 |
| sean-k-mooney | i.e. extend kolla ansibel to deploy zuul for your alongside your openstack | 19:39 |
| clarkb | maybe? I think a big part of the problem is that it requires really domain specific knowledge and info | 19:39 |
| sean-k-mooney | you would still need a refernce thid-party ci repo that you could clone | 19:39 |
| sean-k-mooney | but it might help | 19:39 |
| clarkb | and trying to encode that in an installer for generic stuff is going to lead to friction | 19:39 |
| fungi | yes, for a while there were a bunch of 3pci operators installing software factory's zuul distribution | 19:40 |
| clarkb | installing zuul is almost never the problem | 19:40 |
| clarkb | configuring zuul to run devstack and tempest is | 19:40 |
| sean-k-mooney | which is odd because its basically this https://github.com/SeanMooney/ci-sean-mooney/blob/main/playbooks/tempest/run-tempest.yaml | 19:40 |
| fungi | opendev itself avoids providing a reusable zuul distribution based on its configuration because 1. our situation is fairly niche, and 2. we don't have bandwidth to support people trying to install copies of our environment | 19:40 |
| sean-k-mooney | ok its more then that but that the core of it | 19:40 |
| clarkb | you also have to remember that a lot of these people have never run a ci system before or run devstack or run tempest and may not even understand yaml or python | 19:41 |
| gmann | devstack, tempest configure/run can be referred from any of the upstream job, we have a lot of different configuration example there | 19:41 |
| clarkb | because of the way we push the requirement on people integrating hardware stuff there is a lot of "this is openstack" that they have to learn | 19:42 |
| clarkb | running a service is often not the hard part | 19:42 |
| fungi | the zuul community provides a quickstart example based on its published container images, but similarly doesn't have the bandwidth needed to support people who want to figure out how to shoehorn that example into something that will satisfy openstack teams with third-party testing requirements | 19:42 |
| sean-k-mooney | ya i have been on both side of that. it was a lot to learn and a lot to invet in at the time | 19:43 |
| sean-k-mooney | so the only time projects require third paty ci si if it cant be tested in the first party ci | 19:44 |
| sean-k-mooney | for netuon the escpae hach has alwasy been jsut implement the feture in any of the intree backend to test the api/common bits and then you can supprot it in yoru third party backend without any future testing | 19:45 |
| sean-k-mooney | ironic and cidner have the same escape hatch to a degree | 19:45 |
| sean-k-mooney | if you can implement it in the lvm driver and yoru our out fo tree vendor driver then thats enough to add the api feature | 19:46 |
| fungi | gmann: often times the challenge is figuring out which upstream projects need to be added to the zuul tenant config. even just to run a base devstack job you need dozens of different openstack repos | 19:46 |
| sean-k-mooney | yep i actully have that mroe or less | 19:47 |
| fungi | and if you don't get them all added, all you see is zuul config errors because of the required-projects not being found | 19:47 |
| sean-k-mooney | https://paste.opendev.org/show/bGF7XuiXQcstj3hIgDGl/ you dont need anything after line 52 | 19:48 |
| fungi | so "just reuse the job definition in devstack" isn't nearly enough guidance when you're starting from scratch (of course it's easy in opendev because all the projects are already in the tenant) | 19:48 |
| sean-k-mooney | although that actully whiy i defed my own minimal devstack jobs | 19:48 |
| sean-k-mooney | the upstream ones pull in a lot of openstack by default | 19:49 |
| sean-k-mooney | the - include: [] is actuly pretty imporant too which is non obvious | 19:50 |
| sean-k-mooney | if you dont want to have to keep adding more an more projects you need to make thigs aviable without importing there jobs | 19:50 |
| * gouthamr had to step away | 19:50 | |
| sean-k-mooney | i could not import jobs form temept of tempest-lib (i cant remmeber which) because it ended up pulling in a bunch of tempet plugins | 19:51 |
| gouthamr | sean-k-mooney: hey thanks for the feedback on the summaries/minutes etc.. i'm glad the recordings are helping.. i'll attempt to overshare and repeat context where appropriate.. | 19:51 |
| sean-k-mooney | gouthamr: so far i have only really looked at them for thing like the eventlet/awaitlet discussion | 19:52 |
| gouthamr | all this brainstorming is useful to someone starting on this path i presume.. /me wouldn't relate because he hasn't attempted to get a third party CI job with zuulv3.. but | 19:52 |
| gouthamr | it helps to know who all have tried/have ideas | 19:53 |
| gouthamr | in cinder in the past, we directed folks to software factory.. because we had a bunch of rdo contributors, and RDO had a lot of "third party" testing around tripleo | 19:53 |
| sean-k-mooney | so for better or worse i used to use deploying zuul/ci as a way to learn new things | 19:54 |
| sean-k-mooney | i did it by hand many many years ago, then when i neede to larn ansible i did a horbiale job automating it and did the same the first time i was tryign to learn how k8s worked | 19:54 |
| gouthamr | directing to software factory eased hardware, installation and other setup questions that were coming our way.. so i know a number of third party CI maintainers use this route.. if we're open to that, we can ask for an opinionated document too.. | 19:55 |
| clarkb | just remember that installing the ci system is not the hard part | 19:55 |
| clarkb | what needs to be documented is the next step of configuring the project and jobs to execute to achive the requirements openstack has | 19:56 |
| gouthamr | ^ it isn't, clarkb .. but it was probably easier to be told what to do to someone that has never seen zuul.. | 19:56 |
| gouthamr | yep; i agree | 19:56 |
| sean-k-mooney | i think what would help is creating a seperate repo with a very basic zuul/nodepool config and miniaml set of jobs that deploy devstack with its default seting | 19:57 |
| sean-k-mooney | i.e. a ci in a box | 19:57 |
| gouthamr | we've asked people to extend base jobs from manila-tempest-plugin.. and we occasionally get rebuked when making changes there without intimation | 19:57 |
| gouthamr | but, it'd be nice to spell out how this config works (alongside the pipeline config) | 19:57 |
| sean-k-mooney | you can extned jobs without doign that in the same repo | 19:58 |
| gouthamr | sean-k-mooney: ++ | 19:58 |
| gouthamr | sean-k-mooney: i think we had variations of that | 19:59 |
| sean-k-mooney | i have not done the zuul demo in like 4 years but this is effectly what you have to update your zuul config too https://termbin.com/t66i | 19:59 |
| gouthamr | a cinder forum session in the (recent) past on the topic: https://www.youtube.com/watch?v=kvHsZkWxgCc | 19:59 |
| sean-k-mooney | the bit that will catch out first timers is the connecto to opendev shoudl be called [connection opendev.org] | 19:59 |
| sean-k-mooney | if your tryin to reuse the upstream jobs we unfortunetly use the fully qualifed names of repos in required projects in some places | 20:00 |
| sean-k-mooney | i.e. opendev.org/openstack/nova instead of openstack/nova | 20:00 |
| sean-k-mooney | so if you call the conenction "upstream" or "openstack" taht is really hard to debug | 20:01 |
| sean-k-mooney | the sad thing is whil i proably have all the knowladge to help set up a third party ci i just dont have time to help someone else do it | 20:03 |
| gouthamr | :( noted.. | 20:05 |
| sean-k-mooney | i used to have my own third pary ci running on my home cloud in a ubuntu vm deployed on my home openstack. but i reinstalled that host about 2 months ago, it had been power off for about a year before that however | 20:08 |
| gouthamr | what were you testing? | 20:08 |
| sean-k-mooney | honestly not much the plan was to test a specific veartion of nova with ovs/dpdk and multiple numa nodes | 20:10 |
| sean-k-mooney | which we coudl not fully do upstream | 20:10 |
| gouthamr | ah ack | 20:10 |
| sean-k-mooney | i wanted to replace the intel nfv ci which got shutdown right after i left intel | 20:11 |
| gouthamr | i asked because one of the other things people seemed to ask was around how to connect their devstack (on a nodepool test node) to the hardware they were testing with (i.e., a storage system).. doing a static mapping might be okay.. but, the rate of changes in openstack, people like to maintain a bunch of resources they can hand out to their tests.. | 20:12 |
| gouthamr | i was wondering if you faced a similar problem | 20:12 |
| sean-k-mooney | we actully have that problem kind of down stream but yes i have hit that in other contexts | 20:13 |
| sean-k-mooney | the zuul answer is to use semephors when you have a limited resouces | 20:13 |
| sean-k-mooney | but the longer answer is it depnedn on how you can shard it | 20:14 |
| gouthamr | what people do today - have some ansible pre-playbooks to do some prep work and post work; or use a database of available resources some way integrated.. or they've hacked nodepool to create "Device Manager" that works alongside the test node creation | 20:14 |
| sean-k-mooney | right so one of the things i have done in the past is have a job that runs under a semepshre to allocate the resouce and return it as a zuul output which then pauses and will do the cleanup at the end | 20:15 |
| sean-k-mooney | so you have one job providion a host and a second job use that host | 20:15 |
| sean-k-mooney | downstream we have 1 or 2 stoage applances i.e. a netapp san | 20:15 |
| sean-k-mooney | there we can share it between jobs provided each job uses unique host names for teh vms that use the storage and provide we limit the total concurrent usage | 20:16 |
| gouthamr | ^^ ++ yeah, common problem | 20:16 |
| sean-k-mooney | effectivly it comes down to haveign a resocue provider jobs and a consuming josb in most cases | 20:17 |
| sean-k-mooney | the provider handels the allocation and cleanup and you depend on it form the consuming job | 20:17 |
| sean-k-mooney | zuul has tools to help with this to some degree but its largely a build your own worklfow | 20:20 |
| gouthamr | +1, maybe a worthwhile improvement to nodepool since it is a common pattern.. | 20:24 |
| sean-k-mooney | i was trying not to go there but yes you could sort of abuse noodpool and the static/meta provider ot model this | 20:26 |
| sean-k-mooney | really what yoru talkign about is a way to declear and inventory of consumable <thing> and having a way to hand those out to jobs | 20:27 |
| gouthamr | yes | 20:27 |
| sean-k-mooney | as i said the way to do that nativily today is have a jobs that has a playbook that claims the thing, use zuul output to reorced the clamind one and pause until the depnedt jobs are done and then releases it | 20:28 |
| gouthamr | ^ yes; we've been suggesting people do this.. will get this added to the doc we're going to write | 20:28 |
| sean-k-mooney | that does requrie an external invetnotry system however to do the allcoation and deaclloaiton | 20:28 |
| sean-k-mooney | but its workable if you knwo what your doing | 20:29 |
| sean-k-mooney | and that the probelm that a non trivial thing to explian and not somethign peopel are likely to come up with on there own | 20:29 |
| sean-k-mooney | im sure most peopel dont know about https://zuul-ci.org/docs/zuul/latest/config/job.html#attr-job.dependencies | 20:30 |
| sean-k-mooney | and how to use it with https://zuul-ci.org/docs/zuul/latest/job-content.html#return-values to achive that | 20:31 |
| sean-k-mooney | a concreate exampel fo how this is useful is in our openshift-operator based installer we ahve a content-provider job which create a docker registry and build all the require contianers, then we have tempest job that uses the operator to deploy the contianer form the content provider and run tempest | 20:33 |
| sean-k-mooney | the tempet jobs all depend on a singel content provider job in the build set | 20:33 |
| sean-k-mooney | down stream i had a job that use a baremetal provisionign system to install an os on a specific host and a scond job that used nodepools static provider to then deploy on that static host | 20:34 |
| sean-k-mooney | so that powerfull but not easy to use to build larger jobs out of | 20:34 |
| sean-k-mooney | anywya time for dinner o/ | 20:35 |
| sean-k-mooney | by the way there is proably a btter way to do that with https://zuul-ci.org/docs/zuul/latest/config/job.html#attr-job.provides and https://zuul-ci.org/docs/zuul/latest/config/job.html#attr-job.requires | 20:36 |
| sean-k-mooney | so that you could have 1 provider per consumable thing and have any of them provide it if they are free | 20:37 |
| sean-k-mooney | btu i never figured out how to express that | 20:37 |
| gouthamr | sean-k-mooney: thanks for sharing these thoughts/links! | 20:39 |
| sean-k-mooney | sorry for all the noise in the tc channel | 20:40 |
| sean-k-mooney | i was plannign to check in on the watcher topic but thats not tonights problem | 20:40 |
| gouthamr | :) not sure anyone minds.. but its a spillover from the meeting topic | 20:40 |
| gouthamr | sean-k-mooney: yeah, lets wait on the launchpad answers query; and bauzas plans to do a personal reach out too.. | 20:41 |
| gouthamr | (to david tardivel if you didn't catch up on that in the meeting notes) | 20:41 |
| spotz[m] | Apologies all forgot to mark myself absent for today, at OpenShift Commons today and Kubecon the rest of the week if anyone else is about | 21:00 |
| gouthamr | ack spotz[m] | 21:16 |
| opendevreview | Goutham Pacha Ravi proposed openstack/governance master: Resolve to adhere to non-biased language https://review.opendev.org/c/openstack/governance/+/934907 | 22:23 |
| opendevreview | Goutham Pacha Ravi proposed openstack/governance master: Resolve to adhere to non-biased language https://review.opendev.org/c/openstack/governance/+/934907 | 22:30 |
| opendevreview | Goutham Pacha Ravi proposed openstack/election master: Clarify preference on election official neutrality https://review.opendev.org/c/openstack/election/+/934908 | 22:41 |
| gouthamr | bauzas: when you see this, please join #openstack-election | 22:50 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!