Tuesday, 2023-06-06

opendevreviewKe Niu proposed openstack/constellations master: setup.cfg: Replace dashes with underscores  https://review.opendev.org/c/openstack/constellations/+/88521401:55
fricklerknikolla: o.k., what do we need to do in order to proceed? some formal resolution? or just you telling me "go ahead"?04:56
fungiin the past i've just rough taken consensus from tc members or approval of the tc chair as sufficient11:54
knikollafrickler: you have my go ahead13:48
opendevreviewMerged openstack/governance master: Add cinder-nfs charm to OpenStack charms  https://review.opendev.org/c/openstack/governance/+/87950215:10
opendevreviewMerged openstack/governance master: Clarify expectations on keeping Python versions  https://review.opendev.org/c/openstack/governance/+/88215415:13
opendevreviewMerged openstack/governance master: Add openstack-helm PTLs missing irc nick name  https://review.opendev.org/c/openstack/governance/+/88475715:18
fricklerknikolla: thx, I've started with the first batch, will continue later or tomorrow16:35
fricklersadly I cannot update the topic on this stack after merging, will have to take more care for the next ones https://review.opendev.org/q/I6cfa0f392b10edf5e086b130606ba079a651c2a116:35
frickler10 config errors down, 205 to go16:36
knikollatc-members: reminder that meeting is in ~1 hour and this is the one on video call. 16:55
knikollafrickler: awesome, thanks! 16:55
spotz_knikolla: Thanks for the video reminder!17:20
opendevreviewBrian Rosmaita proposed openstack/governance master: Upstream investment: translations infrastructure engineer  https://review.opendev.org/c/openstack/governance/+/88537917:52
jamespageknikolla: yep thanks for the reminder about the video call17:54
jamespageawesome - zoom still seems to be working today17:56
knikolla#startmeeting tc17:59
opendevmeetMeeting started Tue Jun  6 17:59:51 2023 UTC and is due to finish in 60 minutes.  The chair is knikolla. Information about MeetBot at http://wiki.debian.org/MeetBot.17:59
opendevmeetUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.17:59
opendevmeetThe meeting name has been set to 'tc'17:59
knikolla#link https://us06web.zoom.us/j/87108541765?pwd=emlXVXg4QUxrUTlLNDZ2TTllWUM3Zz0918:00
knikolla#topic Roll call18:00
rosmaitao/18:00
gmanno/18:00
spotz_o/ logging in18:00
knikollao/18:00
jamespageo/18:10
JayFo/ omw18:10
knikolla#topic Follow up on past action items18:12
knikolla#link https://etherpad.opendev.org/p/tc-2023.2-tracker18:12
knikolla#action knikolla To fix link redirect to release from docs18:13
knikolla#topic Open Infra Summit and PTG in Vancouver, next week18:13
knikolla#topic Gate health check18:21
gmannthis is something we need to spread or attract more members to do this #link https://docs.openstack.org/project-team-guide/testing.html#project-gating18:33
fungibeing "the openstack job fixer" has been the #1 source of burnout for long-term members of our community18:35
gmannagree18:35
clarkbI'm not able to dial in but this is sort of why I wondered if a project management type role might be somethng to investigate18:35
clarkbthey could help coordinate efforts across teams/groups ratherthan doing it all themselves and maybe that would help spread out the burnout factor18:36
fungiif we take away the ability for random contributors to make recheck comments, they'll just rebase or push trivial new revisions instead18:37
fungilaziness finds a way18:38
JayFclarkb: nobody has a stick. TC doesn't. PTL doesn't. We can have someone who says "you, go fix the gate", but we have no method whatsoever for making those people -- or in many cases the bosses who set the unreasonable deadlines they are trying to meet -- dedicate their time where we want.18:38
JayFThis is where I struggle, it seems like in many cases saying "work on the commons instead of $interestingThing" may lead to loss of that contributors' effort rather than redirection of it18:39
clarkbJayF: sure, but I don't think a stick is necessary. If such a role produces benefits people will go along with it.18:39
JayFclarkb: IMHO, the root cause of the problem is that the benefits of contributing to the commons is mostly invisible, or difficult to track18:40
clarkbits not about forcing people to do anything butkeeping track of where the issues are and pointing people in a common direction towards betterness18:40
JayFMy hypothesis is that such an effort would lead to the same cabal of people going in the pointed-direction while the same disengaged group continues working in their own direction18:40
clarkbThe problem today is that no one or no group exists for that so when eg ironic tests start timing out before anyone even does the most basic debugging they show up asking infra to fix it18:42
clarkbessentially the infra team is acting as a poor proxy for this role/task today and I think there is a more efficient and productive way to do it18:42
knikolla#link https://etherpad.opendev.org/p/tc-leaders-interaction-2023-vancouver18:43
fungiclarkb: it's because they think we have a magic "make it work" switch18:43
gmannthanks 18:43
knikolla#topic Broken docs due to inconsistent release naming18:45
knikolla#topic Open Discussion and Reviews18:45
rosmaita#link https://review.opendev.org/c/openstack/governance/+/88537918:47
fungi#openstack-i18n18:48
fungicome join us!18:48
fungitonyb also replied to the cinder eol thread a few minutes ago18:49
fungiagain i'll float the proposal that we rename "extended maintenance" to "community maintenance" or maybe just "unmaintained" (and get rid of the separate distinction for the unmaintained phase we have now)18:51
fungibut like i said in my reply, if nobody from the distros is actually proposing patches and volunteering to be the reviewers on those branches, i don't think we've succeeded in what we set out to do with them anyway18:52
fungithey're the first to object when we say we want to eol a branch, but i almost never see them propose or review changes for them18:56
rosmaitafungi: i think you're right about that19:03
rosmaitadoes anyone know if there's a list of forum etherpads, or even just a URL-schema most people are using?19:04
tonyb@diablo_rojo is collecting one ATM see recent list post19:05
gmannrosmaita: this is page we might have but not currectly 19:05
gmannhttps://wiki.openstack.org/wiki/Forum/Vanvouver202319:05
rosmaitafungi: also, i imagine everyone you know has pointed this out to you:  https://thehill.com/policy/equilibrium-sustainability/4034986-fungi-may-offer-jaw-dropping-solution-to-climate-change/19:05
knikolla#endmeeting tc19:05
opendevmeetMeeting ended Tue Jun  6 19:05:44 2023 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)19:05
opendevmeetMinutes:        https://meetings.opendev.org/meetings/tc/2023/tc.2023-06-06-17.59.html19:05
opendevmeetMinutes (text): https://meetings.opendev.org/meetings/tc/2023/tc.2023-06-06-17.59.txt19:05
opendevmeetLog:            https://meetings.opendev.org/meetings/tc/2023/tc.2023-06-06-17.59.log.html19:05
knikolla#endmeeting19:05
tonybOne thing that confuses me is that the state of the branches down to train wouldn't have changed the amount of work for the 2023-2088 CVE19:06
rosmaitatonyb: how do you mean?19:06
fungirosmaita: yes, we're doing great, slimy things for the environment19:06
tonybrosmaita: train is the base for RHOS-16, so even if the upstream branch was EOL, RH (you and co) would staill have backported it to train19:07
fungiapparently the backport rh used wasn't something that would have been considered an acceptable stable branch change, as i heard it explained19:08
tonybHmm okay.  That's a little surprising19:09
rosmaitabasically, what fungi said19:09
fungihaving seen some of the things backported in rhel over the years, i don't find that in the least surprising ;)19:10
tonybfor the recorded I'm totally fine with updating the current stable policy as we've largely shown that the existing one doesn't work.  I just want to make sure we do it right for all projects19:10
fungiwhere we likely need to strike a balance is between letting projects who *want* to support stable branches for longer do that, while not having branches open indefinitely because projects are so under-staffed there's nobody to ask to have them made eol19:12
knikollaperhaps an opt-in vs opt-out for em?19:12
fungithat's a huge problem these days, and the main reason most projects have their branches in em until someone mass-eol's ancient branches across the entirety of openstack19:13
fungithey're not being used in most cases, but the only driver to close them out is bulk cleanup of last resort19:13
JayFI'm going to be honest, regardless of branches supported and/or for how long, I don't think the EM distinction holds any value now whatsoever19:13
fungiJayF: would it make any difference if we called those "unmaintained branches?"19:14
JayFfungi: IMO, it should be more binary: maintained or EOL'd19:14
knikollafungi: as a minimum I think that would be a net improvement. 19:14
fungior is it that having the branches open at all is a problem regardless of how we try to say we're not officially taking responsibility for them?19:14
JayFfungi: because I don't think it's ever likely, in reality, that the promise of EM will be found in reality; primarily because as long as the people who developed $oldVersion are still around, it's going to be difficult to get them to ignore it as it rots19:14
JayFfungi: so I'm saying, branches stay open/maintained as long as the projects want them to /are possible (with maybe even a default-EOL date to avoid the scenario you lay out)19:15
JayFbut that actual stage, the "supported but not by everyone" is just ... not that way in reality, and we're better off being able to keep performing releases, and just call it 'maintained' until it's not19:15
tonybJayF: it used to be binary and then operators and vendors wanted .... what we have now (more or less) so we built it with the expectation that " ... and they will come" only they didnt really19:16
fungiso the suggestion is switch the coordinated em transition to be the new coordinated eol point, but allow projects to request some repositories have those branches held open for another cycle (and then renew that request each cycle if they're still committed to maintaining them)?19:17
JayFfungi: something that general shape, yeah. I'm not sure that's exactly it, but that is the direction I'm thinking19:17
fungicool19:18
JayFfungi: I think the complexities come with cross-project interactions ... if Ironic EOL's version X, and nova doesn't, how does nova continue to test that ironic integration19:18
tonybYeah.19:18
JayFthose are kinda the problems that worry me about that shape, but AFAICT they exist now19:18
fungijobs need to be able to switch to installing releases or checking out tags instead of branches19:18
tonybshould we see if there is room for a lareger discussion at the forum/ptg? rather than hijack the cinder room/table?19:18
dansmithwell, it's worse if a library has eoled and nova depends on a change in that library to fix itself19:19
fungidevstack already does this for things that eol "early" while others are still in em19:19
tonybdansmith: Yeah os-brick goes EOL and then nova cant't backport a fix to it :/19:19
fungitonyb: i do think this is something we've traditionally discussed at the forum and its predecessors, yes19:19
dansmithright19:19
JayFdansmith: yeah, that's a good ID of another edge that would be concerning for sure... but all these problems already exist, by evidenced by our numerous examples :D 19:19
gmannIMO, keeping EM just for backport and remove the integration tests at the start itself is way forward otherwise we keep fixing gate even we do not need to19:19
fungicinder's decision is their own of course, and allowed within the current policy, but it hints at the probability that the whole model needs a rethink19:20
JayFI just don't like it when docs lie, and right now our docs about what EM is, and how we manage it, is a complete whopper of a story LOL 19:20
dansmithI'm just saying, it's not only about cases we can't test because some other service doesn't have a branch.. it would require a backport to a library that is eol *and* a release *and* a requirements change19:20
gmannbut how EM of lib work? as they cannot be releases, no update in constraints 19:21
fungiyes, i do think this is overall smoother when everything goes eol at the same time (like it did before em came about)19:21
dansmithexactly19:21
JayFgmann: this is what I'm saying, EM should no longer exist19:21
JayFgmann: we would continue on a model where as long as a project is supported, it can release19:21
JayFif something you depend on needs an upgrade to fix your bug, well, you should've been there to keep your dependency alive if your project cares19:22
gmannJayF: IMO we can keep iit and remove the gate from start itself. a branch can have backport but test it before use. 19:22
fungithough what we generally said was that if a project (library is a vague distinction given the interconnectedness of our projects anyway) goes eol and then something needs a fix in it, just consider that impossible and eol the thing that needed it or stop running that test if possible or whatever the easiest and lowest cost solution is19:22
gmannuntil we keep test (as long as they work) it create difficulties 19:22
fungithe main problem with the eol-once-things-break model is that there's no easy way to provide advance warning of something we can't predict19:23
JayFfungi: as evidenced by email complaints to mailing list about EM branches going away; I think that distinction is overvalued19:24
knikollaMy personal opinion is that I don't think the benefits of EM make it worth all the downsides and difficulties that it brings. 19:24
gmannyeah and always do EOL in coordinated way is easy for everyone but difficult to do19:24
JayFknikolla: lets be specific: do you mean "the benefits of maintaining branches older than 1.5y" or do you mean "EM, the state, as currently designed"19:24
JayFknikolla: I'm trying to draw that line; because I think EM, the state as defined, is silly and bad; but that we can consider ridding ourselves of that state /without/ just killing all branches at 1.5yr mark19:25
knikollaThe benefits of maintaining branches that we don't support. Whatever our definition of support is.19:25
gmannwe have not made a clear distinguish of support here even doc tell it but in practical we are keeping it almost same.19:26
gmannin ML, cinder is mostly raising point of testing cost and the bandwidth it need to get it merge19:27
JayFthis is why there's value in, if we are maintaining the "EM" branches longer, just calling them "maintained" and continue to do release activities19:27
JayFwhen it's dead, it's dead, and goes straight to eOL19:27
JayFthat middle ground of "the branch is there but only for drive-by contributions or stable-dedicated contributors" does not exist19:27
fungii still think scheduling eol is likely to be of benefit to downstream consumers, as long as we publish that schedule19:28
gmannyes19:28
JayFThe OpenStack Integrated Release goes EOL at the 18 month mark. Ironic/Nova/Glance/Others might continue to be supported after that19:28
knikollaFrom a core reviewers perspective, I assume the following are true 1) I feel personally responsible for everything that merges (including EM branches) 2) I do not feel comfortable merging anything without tests (including in EM branches)19:28
JayFjust maybe not as the bundled package with a bow on top, with promises it works together and are tested together?19:28
JayFI don't know if I love that19:29
* JayF == knikolla19:29
gmannI am ok to EM in between but make it more bakports-but-untested so that very minimum work for upstream members19:29
fungijust marking things eol once we suddenly decide they're unmaintainable is likely to present a bigger problem to downstream consumers (we "warn" them that once it enters em it can eol at any time, but that's not the same as setting an eol date and sticking to it)19:29
gmannnot just downstream but for upstream coordinated and testing other dependent service also 19:30
knikollaWe can make a resolution that 1) creates a maximum age for EM branches 2) reduces that maximum age every year until it reaches zero and we have no EM. 19:31
knikollaTo make it less sudden.19:31
opendevreviewBrian Rosmaita proposed openstack/governance master: Upstream investment: translations infrastructure engineer  https://review.opendev.org/c/openstack/governance/+/88537919:32
gmannthis will help but not completely. frustration and time goes on testing failure and complexity to get backport merged due to gate health/complexity of integration testing  19:33
fungialso the vast majority, as we've observed, are simply ignoring those branches and waiting for a forcing factor (bulk cleanup) anyway19:34
JayFknikolla: I would be -1 to a proposal that forces a project to EOL a branch before they are ready to.19:34
fungii think most of them have no expectation there will every be a change on those branches, but have no interest or incentive to close them early19:35
JayFknikolla: if you haven't, I suggest reading fungi's comment at 19:17:12 UTC (at least in my client), this represents something I think is possible19:35
fungiso the default is they stay open and unused until we say they have to get deleted19:35
fungiwhich is fairly inefficient19:35
gmannand backport as much you can even it miss some dependent lib backport19:36
knikollaJayF: I appreciate the feedback. I'm open to the idea of keeping it opt-in for project teams that want that and requiring renewed commitment (by perhaps keeping the gate running)19:37
knikollaThis only applies for projects with few dependencies or no on other projects though. 19:37
fungiso yes, i do think that if projects are allowed to not participate in whatever the new coordinated eol schedule is, they need to not only opt out of the eol batch but also keep opting out explicitly. make the default eol instead of extension19:37
JayFknikolla: yeah, keeping branches open for >18m being opt-in only I think is the change we need to bring that about19:37
dansmithis it not okay to say those opt-in branches would be kept in a separate repo?19:37
fungiand make it so that just because they opted out once, that doesn't mean we forget to ever eol the branch after the mass eol date has passed19:38
clarkbI'd presonally prefer we not fork a bunch of repos. We already did that for packaing and I hate that its something I now have to support until the end of time19:38
JayFRepository is our base level of organization, where ACLs are setup, how we reference things in governance, etc -- I would not be onboard for putting any openstack code belonging to $project into a different repo than $project19:38
fungiextra copies of our repositories is not terribly efficient19:39
dansmitha repo of nova-legacy where those branches are would really help convey the legacy-ness, but okay19:39
clarkbya it makes every gerrit upgrade take longer and replication for new git mirrors take logner and doubles our disk space requirements etc19:40
fungisome of that is down to gerrit's design, it doesn't deduplicate commits between forks19:40
dansmithright, I know gerrit and our job stuff makes that less easy than just github repos19:41
fungiapparently github has its own secret sauce storage backend to make forking a low-resource action19:41
dansmithmight also be an opportunity to say "and these things no longer run tests because they've been moved out" :)19:41
knikollaCould we have a "openstack-archive" that isn't in gerrit but just git? For those archived repos in a read-only manner? 19:42
dansmithyeah that ^19:42
JayFYou'd create so much churn for downstreams; anyone consuming the old release from git would have to make positive changes19:42
gmannbut how read-only help ? no backport ?19:42
dansmithit would be like the archive.ubuntu.org mirrors, where they're still there, people can install from them as needed, but it's clear they're not getting updates anymore19:42
JayFwe're literally adding a hurdle of the type a user specifically complained about on the list when a train EM branch we eol'd19:42
dansmithyep, but a very clear one,19:43
dansmithand once you do, it's in archive forever19:43
knikollagmann: this is just for the complaint about branches disappearing. For actual EM with testing and backports and merging it would have to be in Gerrit. 19:43
dansmith(but it's still a branch for branch-liking people)19:43
gmannI think if we have branch somewhere in upstream people expect the backport19:44
knikollagmann: Not if it's clearly labeled as "openstack-archive", I hope. 19:44
gmannbut keeping all branches as archive is not bad idea but that does not solve EM issue we have currently 19:45
dansmithwell, you could even push backports to -archive if you want, but the thing that matters to me is that it becomes a very clear distinction from our supported branches in the main repo for the project19:45
knikollaAnd for projects that are doing EM (not in openstack-archive), that expectation would actually be real, since they're going through the hassle of keeping the gates running and proving that they want to keep maintaining that branch. 19:45
clarkbdansmith: how is that any different than having a branch called archive-train?19:45
dansmithI understand it's not super easy, so that's cool, but doing that would address multiple problems I have with it (I both think keeping branches that look maintained around is bad, but I also hate deleting branches and replacing with tagas)19:46
clarkbI think to me it conveys the same message but one happens to be far more efficient and easier to deal with19:46
dansmithclarkb: yeah, archive/train would be a lot better than what we have now which is stable/train honestly19:46
dansmithin terms of messaging I think19:47
gmannyeah *stable/* is confusing for EM for sure 19:47
dansmithI understand it's inefficient for non-human reasons, so that's fine, I just think a separate archive namespace makes a much clearer signal about what these things are19:47
dansmithit sounds like separate repo/namespaces is a non-starter anyway, so we can just drop it19:48
gmannthere is no way to know what is EM until you looks into the release page19:48
knikolla++, stable/<release> works for me. 19:48
knikollaerr, archive/19:48
dansmithhowever, to the CVE point, I think we really do need to stop maintaining things in archive/ or archive- or delete the branches if that's what we have to do.. because backporting minor fixes makes it look supported, regardless of what we say in the policy19:49
rosmaitayes, that's exactly what cinder team is worried about19:50
rosmaitait looks maintained because there's some activity, but it's dangerous to actually use it19:50
dansmithI *do* think separating that activity into a separate repo changes the implication of that activity a little, but i definitely needs to stop at some point19:51
gmannwho issue is from what implementation/expectation of EM process when it was created in vancouver/sydney summit19:51
gmannwe always expected there will be a super hero team from somewhere for these EM and not the upsteam core team of that project :)19:52
knikollaTo summarize, my point is that JayF shows that there is a desire from some teams to actually "extend maintenance" of a branch in a process that involves clearly opt-in (that needs to be renewed) and clearing a bar of keeping gates and tests running. And for all other teams that can't maintain that level of activity and commitment, push them to archive. But also don't nuke the repo for archival purposes.19:52
gmannif it is same core team has to do all work then it is becoming same as supported and no clear difference 19:53
tonybbut then we also have consumers complaing that we EOL too soon19:53
knikollagmann: exactly. In my view this is more about giving teams the chance to support a release beyond the EOL date of OpenStack releases. 19:53
knikollatonyb: perhaps we do, but this is about aligning the expectations to what reality is. 19:54
gmannEOl complain was ok but idea was to allow them to help in the extended support of branches:)19:55
fungiif consumers complain that we eol too soon, they should help fix that by assisting with maintenance of the branches we've still got open first. the em model clearly proved that they won't though19:55
knikollafungi: ++19:55
gmannand it end up on more supported branches with just two different name19:55
gmannfungi: exactly 19:55
fungimy answer to "users will complain about this and yet do nothing to help out" is to ignore them, because they're dead weight19:55
fungiif they want a free lunch, they're welcome to eat what we're serving19:56
knikollaThere is value in letting projects keep repos open if they have the proven ability to, but otherwise I do want to nuke EM branches by default. 19:56
fungiif they're willing to help out in the kitchen, then that's another matter19:57
tonybI guess it will be a super fun cinder session ;P19:58
knikollaIf people complain that we EOL things too soon, we can point them to the various way they can extend that.19:58
fungiwe can also point them to this rather lengthy experiment we ran for extending it, and welcome them to explain how it would be any different next time20:11
opendevreviewBrian Rosmaita proposed openstack/governance master: Upstream investment: translations infrastructure engineer  https://review.opendev.org/c/openstack/governance/+/88537921:06
fungitonyb: i agreed with your forum addition suggestion, you just won't see my reply because gmail says i look shifty23:53
tonybLOL23:54
tonybfungi: sounds good23:54
fungino idea why it randomly accepts and rejects messages i send to people. you'd think it would be consistent, but "Our system has detected that this message is likely unsolicited mail. To reduce the amount of spam sent to Gmail, this message has been blocked."23:54
tonyb:(23:54
fungii sent three other messages to gmail addresses earlier today with no problem23:54
tonybI had that alot until I added DKIM and SPF to my domain23:55
tonybI do see your messages to lists, but google are "less picky" about them23:55
fungiyeah, i'll just fall back on irc. i don't want to cater to the gmail mafia that badly23:55
tonybOkay, As long as Josh sees you're okay with it23:56
fungiyeah, rackspace is far less picky23:56
fungibut also i have out-of-band magic to pester him too, if it becomes necessary23:57

Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!