| *** diablo_rojo is now known as Guest3022 | 00:55 | |
| *** Guest3022 is now known as diablo_rojo | 01:03 | |
| diablo_rojo | Next week is newsletter week! Get your stuff in before the end of the week! https://etherpad.opendev.org/p/newsletter-openstack-news | 01:04 |
|---|---|---|
| gmann | tc-members: need one more review on these https://review.opendev.org/c/openstack/project-team-guide/+/837787 https://review.opendev.org/c/openstack/governance/+/839317 https://review.opendev.org/c/openstack/governance/+/839470 | 02:06 |
| gmann | gagehugo: I commented on the order of patches for osh-doc repo retirement https://review.opendev.org/q/topic:osh-docs | 02:14 |
| gmann | basically we need to merge the 1. end gate patch 2. repo content removal 3. governance patch 3. rest everythings like project removal/requirement/doc site if needed etc | 02:15 |
| gmann | let me know if any query, I recently updated the process to make sure governance patch goes first before project rename which is costly if we need to revert that in any case. | 02:16 |
| *** pojadhav- is now known as pojadhav | 06:44 | |
| *** pojadhav is now known as pojadhav|lunch | 08:21 | |
| *** diablo_rojo is now known as Guest3062 | 09:09 | |
| *** pojadhav|lunch is now known as pojadhav | 09:11 | |
| fungi | we don't rename projects when retiring them though? | 11:16 |
| *** pojadhav is now known as pojadhav|afk | 11:42 | |
| *** pojadhav|afk is now known as pojadhav | 12:45 | |
| opendevreview | Merged openstack/project-team-guide master: Fix the dependency hierarchy in repo retirement process https://review.opendev.org/c/openstack/project-team-guide/+/837787 | 12:55 |
| *** pojadhav is now known as pojadhav|afk | 13:42 | |
| mnaser | fungi: i think maybe gmann was referencing the x/... stuff | 14:50 |
| fungi | i don't follow | 14:57 |
| fungi | we don't rename projects into x/... when they're retired | 14:57 |
| gmann | if anyone retiring the project and moving to x/ namespace like neutron-fwaas case was propsoed | 14:57 |
| gmann | but that did not happen and we are maintaining it in openstack only | 14:58 |
| gmann | #startmeeting tc | 15:00 |
| opendevmeet | Meeting started Thu Apr 28 15:00:09 2022 UTC and is due to finish in 60 minutes. The chair is gmann. Information about MeetBot at http://wiki.debian.org/MeetBot. | 15:00 |
| opendevmeet | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 15:00 |
| opendevmeet | The meeting name has been set to 'tc' | 15:00 |
| gmann | #topic Roll call | 15:00 |
| slaweq | o/ | 15:00 |
| gmann | tc-members meetig time | 15:00 |
| rosmaita | o/ | 15:00 |
| dpawlik | o/ | 15:00 |
| gmann | o/ | 15:00 |
| dansmith | o/ | 15:00 |
| knikolla | o/ | 15:00 |
| jungleboyj | o/ | 15:00 |
| gmann | today agenda #link https://wiki.openstack.org/wiki/Meetings/TechnicalCommittee#Next_Meeting | 15:01 |
| gmann | let's start | 15:02 |
| gmann | #link Follow up on past action items | 15:02 |
| gmann | dpawlik to send the new dashboard communication on openstack-discuss ML | 15:02 |
| gmann | he sent that #link http://lists.openstack.org/pipermail/openstack-discuss/2022-April/028346.html | 15:02 |
| dansmith | already done | 15:02 |
| dansmith | :) | 15:02 |
| gmann | thanks | 15:02 |
| gmann | we will talk about it in separate topic too | 15:03 |
| gmann | gmann to schedule call on tick-tock release note question | 15:03 |
| gmann | I did not do this, I will day | 15:03 |
| gmann | rosmaita: dansmith what time is best for this discussion ? | 15:03 |
| gmann | our next TC meeting is video call on 5th may you want to cover in that? | 15:04 |
| rosmaita | that sounds like a good plan | 15:04 |
| dansmith | sure | 15:04 |
| gmann | ok, I will keep at least half an hour for this or more if less topic | 15:04 |
| gmann | thanks | 15:04 |
| arne_wiebalck | o/ | 15:04 |
| rosmaita | that sounds like plenty of time | 15:05 |
| jungleboyj | ++ | 15:05 |
| slaweq | +1 | 15:05 |
| gmann | and in paralel I am checking with foundation about trademark checks on 'tick-tock' name so we might be ready with that | 15:05 |
| jungleboyj | I thought we were going to try to get away from the term tick-tock ? | 15:06 |
| gmann | we said these are ok but let's check legal things also for name | 15:06 |
| dansmith | jungleboyj: we did? | 15:07 |
| jungleboyj | Maybe that was the Cinder team that was complaining about the name. | 15:07 |
| dansmith | tick-tock has the same meaning elsewhere in the industry, so we get the benefit of not inventing new words | 15:07 |
| dansmith | yes, they did do that :) | 15:07 |
| arne_wiebalck | jungleboyj: I think we only said check if there are concerns since it seems connected to intel. | 15:07 |
| gmann | yeah, if trademark is ok then we will go with these name | 15:07 |
| jungleboyj | Ok ... | 15:07 |
| dansmith | arne_wiebalck: right | 15:08 |
| fungi | wes said he already heard back from legal counsel, but i guess he hasn't followed up with you yet | 15:08 |
| jungleboyj | If they complain again I will explain accordingly. :-) | 15:08 |
| fungi | i'll prod him | 15:08 |
| gmann | jungleboyj: +1 | 15:08 |
| dansmith | if we can't use them, then we can't but if we can, I think it is best to stick to the same terminology | 15:08 |
| gmann | agree | 15:08 |
| rosmaita | we need to hold the next ptg in new jersey so we can all meet at the Tick Tock Diner | 15:08 |
| dansmith | intel failed to trademark 3, 8, and 6, but.. tick-tock, maybe they were successful :P | 15:09 |
| knikolla | ooo i miss in-person ptgs. | 15:09 |
| gmann | next one might be. but not final | 15:09 |
| jungleboyj | Cedar Rapids, IA would work as well. One of our favorite restaurants is call the 'Tick Tock'. | 15:09 |
| * arne_wiebalck checks the menu at the tick tock diner ... | 15:09 | |
| gmann | :) | 15:10 |
| gmann | let's move to next topic | 15:10 |
| gmann | #topic Gate health check | 15:10 |
| gmann | any news? | 15:10 |
| dansmith | yeah, so, | 15:10 |
| dansmith | I've been working on the perf stats collection thing, which has been ... educational | 15:11 |
| dansmith | but I realized that we're generating more than 100k rows of query log, which causes it to roll over, such that comparing two runs to each other is problematic | 15:11 |
| dansmith | I upped the limit to 1m rows and we OOM | 15:11 |
| rosmaita | that's ironic | 15:11 |
| dansmith | so I'm trying other limits, but I'm concerned we'll be adding fuel to the fire here at this point :/ | 15:11 |
| dansmith | I wish we could get query stat counters without logging friggin everything, but it doesn't look like we can | 15:12 |
| fungi | this is sql query logs specifically? | 15:12 |
| gmann | how about going with 'compare with static data' only? | 15:12 |
| dansmith | gmann: that's the problem and what I'm trying to do | 15:12 |
| dansmith | gmann: generating static data from one run where we only have "the last 100k queries, whatever those are" is a different set of data from "the last 100k queries of this run" | 15:13 |
| dansmith | so the numbers change randomly, when they shouldn't | 15:13 |
| dansmith | work fine for tempest smoke, but when you compare two full tempest runs, you get seemingly wide variations because you're comparing different sets of data | 15:13 |
| dansmith | so anyway, | 15:13 |
| spotz | Ack sorry, I'm here! | 15:13 |
| gmann | and that is mainly because of polliing API call? | 15:13 |
| dansmith | gmann: no, I thought that was what was generating the variation, but it's because we're rolling over our logs | 15:14 |
| dansmith | gmann: I mean, there's some variation due to polling as well, but I'm nulling that out for the api counters now | 15:14 |
| gmann | ok. +1 | 15:14 |
| dansmith | we don't need to solve it here, | 15:14 |
| gmann | I remember | 15:14 |
| dansmith | just giving an update | 15:14 |
| dansmith | that I'm trying, and I thought the counter-based approach would be easier because it wouldn't be sensitive to performance variations, but of course, devil in the details | 15:15 |
| gmann | +1, I think this perf data is going to be very important things to check in our CI | 15:15 |
| dansmith | other than that, I think there was some oslo policy thing breaking docs changes? | 15:15 |
| slaweq | regarding rechecks I started "monitoring" them and I prepared simple etherpad https://etherpad.opendev.org/p/recheck-weekly-summary | 15:15 |
| gmann | dansmith: I heared in nova channel about oslo policy break but did not check yet. | 15:16 |
| dansmith | slaweq: what are the numbers, like 5,0? | 15:16 |
| dansmith | gmann: I think whoami-rajat has a patch up, or so I heard | 15:16 |
| slaweq | dansmith: average number of rechecks before patches in that project were merged | 15:16 |
| gmann | ack | 15:17 |
| slaweq | I checked patches merged in last week | 15:17 |
| dansmith | slaweq: naked or non-naked? | 15:17 |
| *** pojadhav|afk is now known as pojadhav | 15:17 | |
| gmann | so two number are like in check and gate pipeline? | 15:17 |
| slaweq | what do You mean by naked? | 15:17 |
| fungi | recheck without a reason included in the comment | 15:17 |
| slaweq | I count basically number of "build failed" comments from zuul | 15:17 |
| dansmith | "recheck" with no reason.. you are just counting all rechecks? | 15:17 |
| dansmith | ah | 15:18 |
| slaweq | ahh, I didn't check them | 15:18 |
| dansmith | interesting that swift is so high | 15:18 |
| slaweq | but in patches which were rechecked most times it was "naked" | 15:18 |
| dansmith | ack | 15:18 |
| rosmaita | our message has not gotten through yet | 15:19 |
| slaweq | next week I will add info about how many patches was merged | 15:19 |
| gmann | I still did not get 0,37 or 5,0 means? | 15:19 |
| slaweq | gmann: 0,37 is average number of rechecks in all merged patches in last week | 15:19 |
| dansmith | average 5 rechecks to merge | 15:19 |
| slaweq | so I checked each patch which was merged and count in each of them how many "build failed" comments were on last patchset | 15:19 |
| slaweq | that many times basically it had to be rechecked before it was merged | 15:20 |
| fungi | 0,37 meaning 37%? | 15:20 |
| fungi | and 5,0 meaning 500%? | 15:20 |
| slaweq | and I count average number of such rechecks in all patches | 15:20 |
| dansmith | fungi: no, I think it means on average 5 rechecks to merge a patch | 15:20 |
| slaweq | fungi: no, 5.0 means that all patches in swift in average were rechecked 5 times to get merged | 15:20 |
| gmann | so for example this 2,75. what 2 donate and what 75 donate ? | 15:21 |
| dansmith | 2.75 | 15:21 |
| slaweq | it could be e.g. 2 patches - one rechecked 10 times and one merged at first try | 15:21 |
| gmann | oh is it . or , | 15:21 |
| dansmith | gmann: put on your eurogoggles :) | 15:21 |
| slaweq | yeah, it should be with "." :) | 15:21 |
| arne_wiebalck | is "build failed" the same as "recheck", or simply close enough? (a broken patch set will also result in "build failed" and then a fixed set is pushed and the build will work, no?) | 15:21 |
| slaweq | it's decimal number | 15:21 |
| spotz | eurogoggles:) | 15:21 |
| gmann | with , I thought it is two different number donating two things | 15:22 |
| slaweq | yeah, sorry for confusion | 15:22 |
| gmann | got it now. | 15:22 |
| slaweq | I will be better with it next week :) | 15:22 |
| slaweq | "eurogoggles" - I like that :) | 15:22 |
| dansmith | arne_wiebalck: build failed and rechecks are not the same for sure, and some people will resolve with a patch push instead of a recheck | 15:22 |
| fungi | yeah, so 0,37 is the same as 0.37 (37%), and 5,0 is the same as 5.0 (500%) like i said | 15:22 |
| dansmith | so swift could be pushing lots of broken patches and be high on the list | 15:22 |
| dansmith | without rechecking | 15:22 |
| arne_wiebalck | dansmith: this is what I mean | 15:23 |
| gmann | yeah | 15:23 |
| slaweq | arne_wiebalck: that's why I count build failed comments only on last patchset | 15:23 |
| dansmith | fungi: I don't see how these are percentages :) | 15:23 |
| slaweq | the one which was merged | 15:23 |
| fungi | rechecks as 500% of patches | 15:23 |
| arne_wiebalck | slaweq: oh, ok, thanks! | 15:23 |
| dansmith | slaweq: ah, that's better | 15:23 |
| fungi | a.k.a. 5x | 15:23 |
| gmann | slaweq: yeah. but there are still more recheck on patches in-progress | 15:23 |
| slaweq | gmann: but I'm checking only build failed in last patcheset on patches which are already merged | 15:24 |
| gmann | ok | 15:24 |
| slaweq | so there are only "good" patches counted | 15:24 |
| slaweq | and final ones | 15:24 |
| arne_wiebalck | slaweq: so it is rather conservative as there might be rechecks before | 15:25 |
| timburke_ | i'm confused about these numbers. the swift patch listed under "Patches with most rechecks from last week" (https://review.opendev.org/c/openstack/swift/+/837036) supposedly has 5 rechecks, but i see only 2. perhaps the non-check/gate pipelines should not be counted? | 15:25 |
| slaweq | timburke_: yeah, now I see that I need to improve it a bit | 15:26 |
| slaweq | it seems it counted build failed (arm64 pipeline) results too | 15:26 |
| slaweq | and that shouldn't be the case probably | 15:26 |
| gmann | timburke_: main idea is not to do recheck without comment or debugging #link https://docs.openstack.org/project-team-guide/testing.html#how-to-handle-test-failures | 15:26 |
| slaweq | when I wrote that script there wasn't that arm64 pipeline yet :) | 15:26 |
| fungi | yes, looks like failures in non-voting pipelines were probably included | 15:27 |
| gmann | slaweq: can we just count the recheck with no comment? I think that is what we target first as part of this work, to educate people on this. | 15:27 |
| dansmith | slaweq: in case it's not clear, we're still very thankful that you're doing this, despite having some feedback about methodology :) | 15:27 |
| fungi | i agree limiting it to check and gate is prudent. failures in pipelines like promote or experimental are noise for this purpose | 15:27 |
| gmann | how many recheck with comment is something different that 'why this project is not so stable or what frequent failure we need to solve' | 15:28 |
| timburke_ | got it. i can work on that. fwiw, the other two failures fall into either repo mirror troubles (the probe test retry limit) or an eventlet deadlock bug (https://github.com/eventlet/eventlet/issues/742 -- i need to bug temoto for a release that includes the fix) | 15:28 |
| slaweq | gmann: I will do something to check "naked" rechecks | 15:28 |
| gmann | timburke_: +1 | 15:28 |
| dansmith | gmann: well I think slaweq was also looking to find which projects are "recheck grinding" patches into the gate that might further make it less stable | 15:28 |
| dansmith | so I think both sets of data are useful | 15:28 |
| gmann | sure, if both we can do that is great. and at the end of month or so we can push it on ML or project saying you are doung 'naked recheck' and 'you are not that stable so need to figure out that first' | 15:29 |
| slaweq | gmann: I will prepare such data for each project for next week | 15:30 |
| gmann | but thanks to slaweq for starting it and its nice start. let's discuss it after meeitng on what we can improve or so | 15:30 |
| gmann | slaweq: thanks again. | 15:30 |
| gmann | on gate, one things is QA decided to drop centos-8-stream testing #link http://lists.openstack.org/pipermail/openstack-discuss/2022-April/028321.html | 15:30 |
| gmann | if any of the job project you know, please remove it or replace the testing with c9s | 15:31 |
| gmann | anything else on gate health ? | 15:31 |
| gmann | #topic Retiring the status.openstack.org server | 15:32 |
| gmann | #link http://lists.openstack.org/pipermail/openstack-discuss/2022-April/028279.html | 15:32 |
| gmann | there are two things are not yet replaced which are proposed for retire 1. #link http://status.openstack.org/elastic-recheck/ 2. #link http://status.openstack.org/reviews/ | 15:33 |
| gmann | for elastic recheck we will discuss in next topic | 15:33 |
| gmann | for review dashbaord, anyone use that? (or was using that as it is broken now) | 15:34 |
| fungi | it's probably been broken for many months, i really have no idea when it started returning empty json | 15:34 |
| fungi | and nobody's replied to my message about it on the ml yet, so i'm expecting it's been entirely unused for a long time | 15:35 |
| slaweq | ups, I didn't even know about it :) | 15:35 |
| gmann | but question is if we need can we fix and make it up? | 15:35 |
| gmann | or no option of that and we need to retire it as no help or infra resource? | 15:36 |
| jungleboyj | We do have Upstream University content that refers to this. So we will need to remove/update that. | 15:36 |
| spotz | I'm not sure we mention rechecks but could add it | 15:37 |
| gmann | fungi: do we have option to bring it up again if anyone want it? | 15:38 |
| fungi | jungleboyj: refers to the reviewday query dashboard, or to something else? | 15:38 |
| jungleboyj | spotz: I think we had information on rechecks. I know we refer to status.openstack.org for monitoring patches being checked and a mention of elastic recheck. | 15:40 |
| jungleboyj | Anyway, not a show stopper in this discussion. | 15:40 |
| fungi | ahh, yes, well elastic-recheck is next on the agenda | 15:40 |
| gmann | yeah, amny porject also using that. we will discuss it next | 15:40 |
| gmann | let's wait for reply on ML but if we do not have option to bring it up if anyone need than I think we cannot do much. and whatever opendev team (fungi ) decide is ok. | 15:42 |
| gmann | moving to next topic? | 15:42 |
| fungi | yeah, the plan outlined in the e-mail was to take it down tomorrow | 15:42 |
| gmann | ok | 15:42 |
| gmann | #topic Communicating the new ELK service dashboard and login information | 15:42 |
| gmann | #link http://lists.openstack.org/pipermail/openstack-discuss/2022-April/028346.html | 15:43 |
| gmann | dpawlik: go ahead on updates and elastic-reheck poosiblities ? | 15:43 |
| dpawlik | so the tripleO project got own version of e-r . They contenerized the service - https://opendev.org/opendev/elastic-recheck/src/branch/rdo | 15:44 |
| dpawlik | Still don't know if TripleO can handle that service or they have enough resources on that server to provide e-r for Openstack. | 15:44 |
| gmann | If they can provde I think that will be great as we do not have e-r service now. | 15:46 |
| dpawlik | by saying own version I mean http://ci-health.tripleo.org/ | 15:46 |
| clarkb | its also maintained in a separate fork of the e-r tool (same repo but different branch) | 15:46 |
| dpawlik | clarkb: exactly, branch rdo | 15:47 |
| gmann | yeah, no idea why but I think we can sync up with tripelo team on this | 15:47 |
| dpawlik | it will be the best idea to get more information | 15:48 |
| gmann | dpawlik: that is ok whatever server it run as long as we get that in OpenStack it is fine. we do not have much resoruce in opendev now so sharing from tripleO or other place is all good | 15:48 |
| gmann | any tc-members to syncup on this with tripleO team? | 15:48 |
| gmann | especailyl redhat folks as they might know eack other | 15:49 |
| gmann | each | 15:49 |
| dpawlik | I will talk | 15:49 |
| gmann | thanks and i will check if any tc-member can join you also. | 15:50 |
| dpawlik | next week I should have full answer | 15:50 |
| gmann | anything else dpawlik on this topic ? | 15:50 |
| clarkb | its probably worth reiterating that the opendev team would like to continue to be able to support these use cases | 15:51 |
| clarkb | the problem we're facing is that it seems its more likely for projects to go and do it themselves and not contribute to the commons | 15:51 |
| spotz | I can probably help | 15:51 |
| gmann | clarkb: you mean ELK service or e-r specially ? | 15:51 |
| clarkb | I think it is a bug that tripleo forked the tool and did their own thing rather tahn work in the commons. But that ship has sailed for this particular insance | 15:51 |
| clarkb | gmann: both | 15:52 |
| gmann | spotz: great, please coordinate with dpawlik . | 15:52 |
| gmann | clarkb: I am confused. you mean if anyone comes and maintain them in opendev then or going back the dicission of shuting down those and with current resource opendev team can do? | 15:53 |
| clarkb | gmann: I mean a year ago we identified shutdown as a risk and said we need help. Since then tripleo forked the tools and has stood up a completely separate instance of things ignoring the commons which have now been shutdown and partially replaced with opensearch | 15:54 |
| fungi | gmann: if those people joined the opendev team we'd have capacity to do more | 15:54 |
| clarkb | right if when we said "we need help" we got help instead of the tool getting forked then we are in a very different situation today | 15:54 |
| clarkb | I think the ship has sailed and we should move on. But I don't want the impression to be with the next thing that we don't wnt help | 15:54 |
| gmann | ok. so its same situation and the reason you have to shutdown | 15:54 |
| clarkb | our position is still that help si what we need as the priority | 15:55 |
| clarkb | we're doing our best otherwise | 15:55 |
| gmann | I agree more coordinatrion from tripleo can be good here | 15:55 |
| dpawlik | from the other topic: it would be good to know what visualization can be created that will be helpful for developers. I made simple one, but they should be replaced. I will write an email in few days | 15:55 |
| gmann | but may be opendev team need to brainstorm how to get that help from projects and why they do not do | 15:55 |
| gmann | dpawlik: +1, sure | 15:56 |
| gmann | dpawlik: I will keep this topic in agenda and we will continue discussion on e-r | 15:56 |
| gmann | moving next | 15:56 |
| gmann | #topic Open Reviews | 15:56 |
| gmann | #link https://review.opendev.org/q/projects:openstack/governance+is:open | 15:56 |
| gmann | I will list quick one here | 15:57 |
| gmann | I checked and most of them are good to go. but we need review on FIPS goal milestone #link https://review.opendev.org/c/openstack/governance/+/838601 | 15:58 |
| gmann | please check | 15:58 |
| gmann | that is all from my side. next meeting is on video call. I will paste the link in wiki. | 15:59 |
| gmann | thanks all for joining | 15:59 |
| gmann | #endmeeting | 15:59 |
| opendevmeet | Meeting ended Thu Apr 28 15:59:33 2022 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 15:59 |
| opendevmeet | Minutes: https://meetings.opendev.org/meetings/tc/2022/tc.2022-04-28-15.00.html | 15:59 |
| opendevmeet | Minutes (text): https://meetings.opendev.org/meetings/tc/2022/tc.2022-04-28-15.00.txt | 15:59 |
| opendevmeet | Log: https://meetings.opendev.org/meetings/tc/2022/tc.2022-04-28-15.00.log.html | 15:59 |
| slaweq | thx | 15:59 |
| slaweq | o/ | 15:59 |
| dpawlik | thanks | 15:59 |
| jungleboyj | Thank you! | 15:59 |
| spotz | Thanks gmann, everyone | 16:00 |
| clarkb | gmann: fwiw I'm really not sure what to do at this point. The e-r situation is particularly frustrating because tripleo folks asked to take on e-r maintenance/ownership and we thought "this is great people are helping" but one of the firs tthings they did was fork and ignroe everyone else | 16:00 |
| clarkb | In this particular situation we thought we had done everything right. We asked for thelp. People showed up to help. But turns out they were not interested in helping existing users only themselves | 16:00 |
| gmann | clarkb: yeah that is the key here. why projects are not helping in 'commons' and more interested to jsut solve their use case. | 16:02 |
| gmann | may be they have no energy to make things common and help everyone and just sovle their things and thats it. less contributors is the key and root cause of these problems | 16:02 |
| clarkb | yes, and the trick is convincing people that solving it once for everyone is less effort than solving it 20 different ways | 16:03 |
| gmann | I keep bringing it in every baord meeting or where ever i get chance but it is always interpreted as 'company should step up and help' but that is problem we have companies are not helping | 16:03 |
| fungi | some of it is also a catch-22 that we've built robust testing and deployment automation for our services in order to enable us to run things with far fewer people, but contributing to maintaining that has its own learning curve and additional complexities | 16:04 |
| gmann | I think opendev is going to be impacted for this most as we are seeing many services are shuting down as no help. | 16:04 |
| fungi | most of the services we've been shutting down are ones which still haven't been moved to our new configuration management tooling, causing them to be a much larger support burden for our limited pool of people | 16:06 |
| gmann | because I am not sure how many company will say "I will invest in opendev". we either should merge the things back or have some baord level strategy about it. it might be running ok now but after 3-4 years I am seeing this as a big problem for projects. | 16:06 |
| clarkb | and many of those are in that situation because they aren't in the most critical set of services and/or are compliacted to update | 16:06 |
| gmann | and seems no one is much bother about it :) | 16:07 |
| clarkb | without e-r you can still push and land code for example. So it isn't super critical | 16:07 |
| clarkb | a complicated update example is translations/zanata | 16:07 |
| fungi | we separated opendev out of openstack because we weren't getting sufficient assistance even then. basically once hewlett packard stopped funding a bunch of sysadmins to work on things | 16:07 |
| clarkb | since we have to convert tooling and all of the automation at the same time | 16:07 |
| gmann | yeah but we are facing same issue here 'not much help' | 16:08 |
| clarkb | gmann: sure but the point is remerging is unlikely to fix that | 16:08 |
| gmann | that is why I am saying some board level strategy we need to avoid future critical issues | 16:08 |
| mnaser | i wonder sometimes if it feels like velocity of getting work done feels slower overall | 16:08 |
| mnaser | personally, while long term it is better to do it for the "greater" good, it is _always_ a significnatly slower path in my experience | 16:09 |
| fungi | also the services we've been shutting down are not "opendev services" for the most part, but rather "openstack community services" within the tact sig (though the people involved are mostly the same) | 16:09 |
| clarkb | mnaser: yup the difference is you can run a large elasticsearch and kibana and e-r for a decade before needing it shut it down. Vs it burning itself down due to pure planning and management | 16:09 |
| mnaser | unfortunately short term results seems to trump long term planned stable stuff :\ | 16:10 |
| clarkb | if we want to look at this situation in a positive light I think the ELK setup is a great example for why the opendev model works | 16:10 |
| opendevreview | Merged openstack/governance master: Update Dmitriy Rabotyagov email https://review.opendev.org/c/openstack/governance/+/839470 | 16:10 |
| opendevreview | Merged openstack/governance master: Remove oslo's TaCT SIG liaisons https://review.opendev.org/c/openstack/governance/+/839317 | 16:10 |
| clarkb | we invested and the system ran for a decade | 16:10 |
| mnaser | (im not saying its the better option, but i'm saying it's probably what 'feels' better for people in perspective) | 16:10 |
| clarkb | even after it stopped being maintained for about half that time | 16:10 |
| clarkb | something that is rushed to solve the problem quickly is far more likely to have even worse tech debt and be even less stable in the face of maintainers leaving | 16:11 |
| mnaser | i don't disagree with you | 16:11 |
| mnaser | but i think in general humans want things "now" | 16:11 |
| fungi | you can implement short-term solutions and deal with the resulting tech debt if you have a lot of people. when it's a handful, careful planning is really the only solution | 16:12 |
| fungi | the short-term mindset usually goes back to "don't worry, we can always throw more people at the problem" | 16:13 |
| clarkb | If that is the biggest barrier to contributing then I'm not sure we'll find contributors though. Because those of us that know we'll haev to deal with the tech debt in the future won't take on significant quick changes | 16:13 |
| mnaser | i think that's the hard piece that we're stuck in | 16:13 |
| mnaser | i see it across a bunch of openstack projects | 16:13 |
| mnaser | large changes require a ton of planning/overhead/etc so it feels easier to look elsewhere | 16:14 |
| mnaser | but that is only making the problem "bigger" | 16:14 |
| clarkb | basically that mindset means I get to do all the work and that isn't any better than today except that at elast today the problem set is shrinking not growing | 16:14 |
| gmann | yeah, broader vision need more people to work togehter. | 16:14 |
| clarkb | mnaser: i think the end result is the same in both cases its just a different path to getting there | 16:14 |
| mnaser | like an example of this i see is horizon / .. sorry my brain is escaping the other name.. skyline? | 16:14 |
| fungi | early on we implemented a lot of services with limited thought to long-term maintainability because we could always just throw more people at the problem. now we don't have that luxury, so we're being more careful about what we run and how we run it so that we can do so with a skeleton crew when necessary | 16:14 |
| mnaser | i guess an alternative of that is "we'll let you go crazy with it, but we'll shut it down right away if you disappear" | 16:15 |
| fungi | yeah, that's precisely how the elastic-recheck replacement is positioned | 16:16 |
| fungi | here's the systems, go wild. and if you disappear and nobody else steps up to take it over before you're gone, then it's not our problem | 16:16 |
| mnaser | yeah, i think that's a fair approach | 16:17 |
| fungi | however, that also necessitates that such services are not tightly integrated into other more important systems, so that they can simply be turned off if needed | 16:18 |
| mnaser | which means extra work too, yeah.. | 16:18 |
| fungi | the ci-logscraper solution dpawlik put together is a great example of that. we have no direct ties to it in our base jobs (unlike the old logstash-worker/gearman model) | 16:19 |
| fungi | if it goes away on no notice, there's no action required from the opendev sysadmins | 16:19 |
| mnaser | yeah but i think the work dpawlik is did is exceptional, it's a lot more than what someone who wants to come in and "fix up something" migth do | 16:22 |
| clarkb | yup, though it is fairly straightforward to say modify the gerrit theming doing a gerrit upgrade to 3.6.0-rc2 to fix rsa ssh keys si a different story | 16:23 |
| gmann | back to e-r service for Openstack, spotz dpawlik let's check with tripleO team if they can make thatavailable for other OpenStack projects too. if that happen issue is solved. | 16:49 |
| clarkb | one potential issue with that is currently e-r is set up to query a single elasticsearch. But modifying it to run queries against different servers based on a query annotation is probably not too much work | 16:52 |
| clarkb | in your query you could set the server name or url potentially | 16:52 |
| opendevreview | Gage Hugo proposed openstack/governance master: Retire openstack-helm-docs https://review.opendev.org/c/openstack/governance/+/839100 | 17:16 |
| opendevreview | Ghanshyam proposed openstack/project-team-guide master: Explicitly mention the each steps of repo retirement https://review.opendev.org/c/openstack/project-team-guide/+/839807 | 18:02 |
| gmann | tc-memebrs: one more review in this osh-doc retirement https://review.opendev.org/c/openstack/governance/+/839100 | 18:14 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!