Thursday, 2022-03-17

*** pojadhav is now known as pojadhav|lunch09:36
*** pojadhav|lunch is now known as pojadhav|10:12
*** pojadhav| is now known as pojadhav10:13
*** lajoskatona_ is now known as lajoskatona10:31
*** iurygregory_ is now known as iurygregory11:45
*** pojadhav is now known as pojadhav|out14:19
*** tosky_ is now known as tosky14:27
gmanntc-members: meeting on IRC in 5 min from now 14:55
*** diablo_rojo__ is now known as diablo_rojo14:58
gmann#startmeeting tc15:00
opendevmeetMeeting started Thu Mar 17 15:00:25 2022 UTC and is due to finish in 60 minutes.  The chair is gmann. Information about MeetBot at http://wiki.debian.org/MeetBot.15:00
opendevmeetUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.15:00
opendevmeetThe meeting name has been set to 'tc'15:00
gmann#topic Roll call15:00
gmanno/15:00
rosmaitao/15:00
diablo_rojoo/15:00
dansmitho/15:00
slaweqo/15:00
jungleboyjo/15:01
spotz_o/15:01
gmann#link https://wiki.openstack.org/wiki/Meetings/TechnicalCommittee15:02
gmannToday agenda ^^15:02
gmannlet's start15:02
gmann#topic Follow up on past action items15:02
gmannno action item form last meeting15:02
gmann#topic Gate health check15:02
gmannany news on gate?15:02
dansmiththere was a creeping failure in the nova-ceph-multistore job,15:03
dansmithwhich was OOMing mysql, which I hopefully fixed by trimming down15:03
knikollao/15:03
dansmiththat affected a few projects at least15:03
gmann+115:04
dansmithI know we've still got the volume detach failure thing going on with the centos jobs, at seemingly 100%, but those aren't voting anywhere that I know of15:04
gmannyeah, I am able to make rescue server test pass with SSH-able server fix but there are two more test failing #link https://review.opendev.org/c/openstack/tempest/+/83160815:04
dansmithoh I thought all the sshable fixes were merged15:04
dansmithoh right, I remember this one now, nevermind15:05
gmanneven after unrescue server we need to wait for ssh-ready before detach happening15:05
dansmiththis breaks non-centos jobs now right?15:05
fungiwe upgraded gitea yesterday, and this exposed a regression with pip install of git remotes from it which impacted blazar's jobs because they were still configured to try zuul v2 era zuul-cloner workflows15:05
gmannyeah, not all. rescue test was seen as failure in reported bug but there are few more15:05
gmanndansmith: no, they will pass. i did recheck. 15:05
dansmithgmann: oh, then why haven't we merged it yet?15:06
fungii've got a fix in the pipe to address the pip install errors, but also it exposed that their jobs are sorely in need of modernizing15:06
gmanndansmith: I was trying a litlte smart for having active server then rescue/unrescue and then SSH but that did not work15:06
gmanndansmith: It just passed yesterday might so will merge after gate pass :)15:06
dansmithoh okay15:06
gmann*yesterday night15:06
gmannfungi: you mean their jobs also on zuulv2?15:07
jungleboyjOn a related note the Cinder team is starting a renewed effort to get people to not do 'naked rechecks' .15:08
dansmithoh very nice15:08
gmann+115:08
rosmaita\o/15:08
fungigmann: more that their jobs hadn't been touched since the zuul v2 days, so were trying to use the zuul-cloner tool to checkout openstack/nova, and because that's not a thing any longer they were falling back to pip install nova from a git remote15:08
dansmithjungleboyj: I try to shame people when I find them doing that with evidence that it's wrong :)15:08
* jungleboyj isn't surprised15:08
jungleboyj:-)15:08
rosmaitadansmith: we expect no less of you15:09
dansmithbut it works ;)15:09
gmannslaweq had good script to collect recheck numbers. 15:09
slaweqspeaking about rechecks, I prepared some data as we talked last week15:09
spotz_*hides*15:09
gmannyeah, I was coming to that15:09
slaweqhttps://docs.google.com/spreadsheets/d/1zlJixCttF1e7ZSJdZzORfrJWRqsllRDn97FHyTr11e8/edit?usp=sharing15:09
slaweqhere are data for "main" repositories15:09
slaweqand if You want it for each repo in openstack/ there is tar.gz file http://kaplonski.pl/files/openstack_rechecks_data.tar.gz15:09
slaweqI collected data for each repo from openstack from last 365 days15:10
diablo_rojoOh thats a pretty cool visualization. 15:10
gmannvery nice. 15:10
dansmithI'm not really sure what the numbers are though15:10
slaweqthe number there are basically average number or rechecks done on last PS before it was merged15:11
gmannis it like <2 recheck per week for most project? or I am reading wrongly ?15:11
dansmithis this rechecks or just build failures?15:11
jungleboyjdansmith:  Ok, good.  Not just me.15:11
slaweqaverage for every week15:11
dansmithslaweq: ah okay15:11
jungleboyjAh ...15:11
dansmithslaweq: so higher numbers potentially mean recheck grinding to get a patch in?15:11
slaweqdansmith: right15:11
gmannohk so it is per patch not all jobs15:11
slaweqit's per patch and average per week15:12
gmannthen 1 or 2 still high number per patch15:12
jungleboyjSo, on average ever other patch has to go through a recheck to merge.15:12
gmannjungleboyj: yeah15:12
jungleboyjWow.15:12
fungiis that just for gate pipeline failures, or check as well?15:12
gmanngood question ^^15:12
slaweqfungi: both15:12
gmanncheck might have mostly but yeah gate also have recheck 15:13
fungiso given that the patch has to pass once in check and once in gate in order to merge15:13
slaweqI was basically counting "recheck" comments on last patch sets15:13
slaweqof course there may be some patches where rechecks were done "on purpose"15:13
slaweqbut in general I think it's not very common practice15:13
jungleboyjFair assumption.15:13
fungialso note that long patch series and/or depends-on can skew this, since one change failing can cause all the ones which rely on it to also fail15:14
gmannslaweq: ohk so on last PS not all recheck on that commit ?15:14
slaweqfungi: true, it's not ideal metric for sure15:14
jungleboyjOh wow, if it is just the last patch set, then the actual number of rechecks per patch could be higher.15:14
slaweqgmann: yes, I was counting only last patch set as I assumed that if that PS was merged finally, it means it was good15:14
gmannyeah, DNM, testing patch also in that but that is ok15:15
slaweqDNM patches aren't in that metric. I was filtering only merged patches15:15
gmannk, +115:15
fungijungleboyj: but remember that this includes check, so includes the "surely the problem isn't my patch, oh i guess maybe it is?" rechecks too15:15
fungior would if you included patch sets before the final one15:16
slaweqI'm using this script and metric in neutron since some time and even if it's not great it shows us pretty clearly current state of the Neutron CI :)15:16
jungleboyj++15:16
jungleboyjSome data is better than no data.15:16
dansmithyeah I think this probably gives us a good view of how much rechecking needs to happen to get something to land,15:16
slaweqhttps://github.com/slawqo/tools/blob/master/rechecks/rechecks.py15:16
slaweqthat is script15:16
gmannslaweq: and is it lot of data if we collect for all recheck including previous recheck that merge PS?15:16
dansmithbut probably needs a bit more to tell us more than that, like if individual patches are actually rechecked more than the average, etc15:16
gmanndansmith: yeah15:17
dansmithbut as a heartbeat sort of thing, if the graph goes up -> bad15:17
slaweqdansmith: I can prepare some data "per patch" too15:18
slaweqI will need to modify that script but it shouldn't be hard15:18
gmannyeah and we can like ignore patches if just 1 recheck or so if data is more15:18
dansmithslaweq: I'm not asking you to do that, just suggesting, but yeah always nice to have data and more data :)15:19
gmannslaweq: just to make sure it is just 'recheck' not recheck with reason?15:19
gmannor both15:19
slaweqgmann: actually it is counting number of comments like "Build failed" on the last PS15:19
slaweqhttps://github.com/slawqo/tools/blob/master/rechecks/rechecks.py#L15515:19
dansmithI really hate that we've drifted away from "recheck with reason" .. I wish we could encourage better behavior there sometime15:19
dansmith*somehow15:19
diablo_rojoThe same way you should be descriptive with your commit message, you should be descriptive with why you are rechecking. 15:20
dansmithdiablo_rojo: I agree, and I never do naked rechecks, but I'm in the minority15:20
fungi"recheck with reason" has always been optional. people who are inclined to look into and record why they're rechecking something will do it regardless of whether it's expected, and people who don't want to bother will make something up like they did back when we enforced it15:20
diablo_rojoI know I am not innocent when it comes to rechecks without explicitly saying why.15:20
slaweqdansmith: sure, I will do modification to have data "per patch"15:20
spotz_I just kind of assumed it triggered off the commit just being recheck15:20
dansmithand I was shamed for my use of shame, so.. lollipops? :)15:20
slaweqI don't know if for next week but I will do that15:21
* jungleboyj is guilty as well.15:21
dansmithfungi: yeah I know15:21
spotz_My guess is training issue. We see just recheck so we use just resheck15:21
diablo_rojoSure its optional, but it would be better if we made it the majority rather than a minority. 15:21
gmannslaweq: no hurry. I am going to add this recheck script/data in PTG and we can discuss what all data we want to monitor per week in zed15:21
diablo_rojoWouldn't want dansmith feeling lonely, you know. 15:21
* dansmith sobs uncontrollably15:21
gmanneasy data will be per week as we monitor weekly so will be easy to check even for all patches or per patches15:22
slaweqgmann: sure, great idea. I will be more than happy to help with that15:22
dansmithspotz_: it used to be required, but people would just "recheck foo" or "recheck bug 00000"15:22
jungleboyjThere there ...15:22
fungii agree it's a good practice, but it's not a good source of data unfortunately because of the number of people who knowingly pollute it15:22
* diablo_rojo hands dansmith a handkerchief "there there" 15:22
gmannslaweq: thanks for this. 15:22
dansmithmaybe we should try to encourage PTLs to push the better behavior in their teams15:22
jungleboyjdansmith:  That is where Cinder is starting.15:23
diablo_rojoslaweq, yes thanks for the data. I look forward to the per patch info!15:23
gmannsure, how? in TC+PTL sessions or in ML?15:23
dansmithjungleboyj: ack, well, let's try to spread that15:23
dansmithgmann: yeah we could start in the PTG session15:23
gmann+115:23
gmannI will add it15:23
slaweqyeah, in neutron we are trying to do "recheck with reason" too but it's not always easy15:23
dansmithgmann: cool15:23
jungleboyj+215:23
slaweqand I also don't do it sometimes :/15:23
slaweqbut I will try to do better :)15:24
gmann#action gmann to add recheck data topic in PTG etherpad (TC and TC+PTL for awareness)15:24
dansmithslaweq: be the change.. be the change.. :P15:24
gmann+10015:24
rosmaitathis is our advice in cinder: https://docs.openstack.org/cinder/latest/contributor/gerrit.html#ci-job-rechecks15:24
slaweqdansmith: yes sir! :D15:24
dansmith:)15:24
spotz_heheh15:24
dansmith24 minutes in and still on gate, eh?15:24
rosmaitajust putting it out there, because i don't know that we are generating machine parseable comments15:24
jungleboyjIt is our favorite topic dansmith 15:25
dansmithrosmaita: "recheck I don't know but at least I looked" is better to me than nothing15:25
gmannfungi: coming back to blazer issue, do you have link for that/job or know if they are working to fix their side?15:25
fungipriteau is working on it, but it was jobs for blazar-nova specifically15:26
gmannrosmaita: I see them sometime machine generated comment (not recheck) and that annoy me more than anything 15:26
gmannfungi: ok. 15:27
rosmaitai have tried to get out third-party ci to add the appropriate gerrit tag so they don't pollute the comments, but you can see how much success i have had15:27
gmannand as frickler pointed out today I pushed moving l-c job to focal/py38 but there are existing config error in that field which needs to be fixed15:27
fungigmann: an old tools/tox_install.sh in blazar-nova specifically15:28
gmannfungi: I see15:28
gmanngood discussion on gate things today. anything else?15:29
dansmith+100015:29
dansmithvery glad to see the gate getting proper attention15:29
rosmaitaslaweq: nice work on that script, btw15:29
gmanntrue, +10000 :)15:29
slaweqthx15:29
dansmithgmann: you just had to +10x me huh?15:30
gmannyeah15:30
dansmithhah15:30
gmann#topic Z cycle Leaderless projects15:30
gmannonly 1 project adjutant left which we are waiting until March end. we will discuss that in PTG15:31
gmannI will remove it from genda15:31
gmann#topic PTG Preparation15:31
jungleboyj++15:31
gmann#link https://etherpad.opendev.org/p/tc-yoga-ptg15:31
gmann#link https://etherpad.opendev.org/p/tc-ptl-interaction-zed15:31
gmannplease add topic in those etherpad15:31
gmanntimeslots are finalized and I have updatad it on ML as well as in etherpad15:32
funginote that the schedule and precreated etherpad links are now live in ptgbot, so can probably safely start adding overrides if needed. diablo_rojo would know for sure though15:32
knikollathe first one is the link from the yoga ptg :)15:32
gmannI have informed Kubernets steering committee for joining us in PTG15:32
spotz_gmann do we need to bring up Sahara, Magnum, etc there or will it be too late15:33
diablo_rojoI think you can override it now. 15:33
gmannsorry #link https://etherpad.opendev.org/p/tc-zed-ptg15:33
jungleboyjhere we go.15:33
gmannspotz_: in TC+PTL sessions?15:33
spotz_Yeah15:33
*** ykarel is now known as ykarel|away15:34
gmannyeah, we call everyone actually not specific projects15:34
diablo_rojoEverything looks correct at this point. I just need to do the zoom room setup once we are closer but that shouldn't affect other things. 15:34
gmanndiablo_rojo: +1, nice15:34
gmannspotz_: and for less active/broken project like sahara, magnum we can address/ping them separately. I would like to keep TC+PTL sessions to get/give feedback sessions instead of going towards project health checks15:35
gmannif we do project health check many PTLs will not join :)15:36
spotz_gmann ok15:36
gmannspotz_: for magnum I know there are few new cores in last cycle which you can ping.15:36
jungleboyjProbably true.15:36
gmannanything else on PTG?15:37
diablo_rojoPlease register if you havent yet!15:37
gmann+1, i did.15:38
slaweqMe too 🙂15:38
gmann#topic Open Reviews15:38
gmann#link https://review.opendev.org/q/projects:openstack/governance+is:open15:38
diablo_rojo\o/15:38
gmannI need one more vote on slaweq vice-chair nomination #link https://review.opendev.org/c/openstack/governance/+/83317115:39
spotz_voted:)15:39
gmannall other open reviews are good, either waiting for time or PTL +115:39
gmannthanks, that is all from my side today. anything else to discuss?15:39
diablo_rojonice :)15:39
gmannwe have around 21 min15:39
jungleboyjVoted!  Thank you slaweq !15:40
gmannthanks and yes thanks slaweq for volunteer 15:40
spotz_Assuming we have joint leadership in Berlin do we want to do anything separate from that?15:40
slaweqyw, I hope I will learn quickly and be able to help gmann there :)15:40
gmannslaweq: +10015:40
diablo_rojoForum submissions should be opening next week I think15:41
gmannspotz_: I think that is good one to restart. and joint leadership meeting is enough at least for Board interaction 15:41
spotz_Sounds good, I pinged the OPS Meetup folks as we're 10 weeks out and really need to get planning15:41
gmanndiablo_rojo: on Forum sessions, do we need TC volunteer for selection committee like we used to have?15:42
gmannspotz_: +1 on ops meetup.15:42
diablo_rojoI have a few PTL volunteers actually15:42
diablo_rojoSo we are good for OpenStack forum selection representation15:42
spotz_I told her I would if no one else staeeped up15:42
diablo_rojoThat too :)15:42
gmanndansmith: nice, I saw wiki and if i understand correctly requirement is not two TC has to be in selection but it can be anyone from community right?15:43
gmanndiablo_rojo: ^^15:43
diablo_rojolol15:43
gmanndansmith: please ignore15:43
dansmithaheh15:43
dansmithI was like ..uhh15:43
gmannyour both name with d* :)15:43
diablo_rojoYeah it can be anyone from the community just ideally someone in a governance position15:43
diablo_rojoso PTLs are great too15:44
diablo_rojoYeah dansmith, here I thought you were the Forum expert lol15:44
gmann"1 delegate from each OpenInfra Project15:44
gmann2 OpenInfra Foundation staff members"15:44
diablo_rojoyep15:45
gmanndiablo_rojo: may be good to mention that clearly about governance in that15:45
diablo_rojogmann, it says elsewhere in the wiki I believe15:45
gmann#link https://wiki.openstack.org/wiki/Forum15:45
gmann"The TC and UC are best placed to decide which of their members should represent each body...." may be this line can be modified now?15:46
gmannthis is left om previous requirement?15:46
diablo_rojoAh yeah that needs to be updated. 15:46
diablo_rojoI will tweak later today15:46
gmannk, just making sure we do not miss anything from TC which we need to do15:46
gmanndiablo_rojo: thanks 15:46
diablo_rojoI would let you know if we were :) 15:46
gmanngreat. 15:47
gmannanything other topic to discuss/15:47
gmann#endmeeting15:48
opendevmeetMeeting ended Thu Mar 17 15:48:12 2022 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)15:48
opendevmeetMinutes:        https://meetings.opendev.org/meetings/tc/2022/tc.2022-03-17-15.00.html15:48
opendevmeetMinutes (text): https://meetings.opendev.org/meetings/tc/2022/tc.2022-03-17-15.00.txt15:48
opendevmeetLog:            https://meetings.opendev.org/meetings/tc/2022/tc.2022-03-17-15.00.log.html15:48
slaweqo/15:48
diablo_rojoo/15:48
spotz_Thanks gmann and everyone!15:48
jungleboyjThank you!15:48
opendevreviewKendall Nelson proposed openstack/governance master: (WIP) Defining Tech Preview  https://review.opendev.org/c/openstack/governance/+/83420817:59
diablo_rojoSorry I have been so behind on doing that but at least its up before the next PTG? lol18:00
dansmithdiablo_rojo: is the next ptg after this one for sure in-person?18:38
diablo_rojodansmith, thats the tentative plan..pending pandemic etc. We don't have exact dates yet, just likely October. The events staff are working to nail down exact date + location by the summit in Berlin18:47
dansmithokay18:47
diablo_rojojust excited to see us all in person eh, dansmith ? ;)18:48
dansmithnope :)18:48
* diablo_rojo is hurt 18:48
dansmithsorry (not sorry) :P18:49
diablo_rojoIf you were, you wouldn't be you :)18:49
dansmithto be clear, I do want to see you all in person, I'm just not excited about actually doing it (as things are)18:49
gmannI am hoping to get community feedback on this before final call. still situation is not so safe to travel. 18:55
diablo_rojoI highly doubt it will ever be as 'safe' as it was in 2019. Kinda like how everything changed for US travel post 2011. I don't think we are ever going 'back'.18:58
gmannIMO, I understand that marketing events can/needs to be physical but developers event can be virutal or hybrid at least (for this year at least).  18:59
gmannjust my opinion though :)18:59
dansmithdiablo_rojo: I think you mean 200119:03
dansmithwe didn't have zoom in 2002, and I agree, it's just not that important to rush back, IMHO19:04
dansmith2001 was a single event, and getting back to normal did not increase the likelihood of it happening again, so not really the same to me19:05
diablo_rojoLol yes I mean 2001 dansmith lol19:07
diablo_rojoobviously I need a nap or more caffeine. 19:07
jungleboyjI think we are going to have to keep more of a hybrid approach because some companies aren't going to go back to letting people travel.19:08
jungleboyjOr it is going to take a while.19:08
gmannyeah that is another factor we should consider. 19:08
gmannand more than that individual choice. 19:08
dansmithyeah, I think the shrunken budgets aren't going to snap back to normal19:09
jungleboyjAmazingly, Lenovo was on the track of 'Everyone must come to the office' in October of 2019.  Now they are going to a permanent hybrid approach.  They want people to come in at some point during the week but are encouraging everyone to work remote for the majority of the week.19:09
jungleboyjdansmith:  Exactly!19:09
gmannindeed, I am sure companies  budget preparation for 2022FY are not considering much travel.  19:10
fungilots of companies dropped the leases for their office buildings19:11
funginowhere to tell the employees to come back to19:11
dansmithhybrid really puts the remote people at a disadvantage, more so than if everyone is remote too,19:11
dansmithwhich is a concern of mine19:11
jungleboyj++19:11
dansmith(for the ptg I mean)19:11
fungii think it should be possible to make sessions less convenient for the people attending in person, in order to put them on an equal footing with those attending remotely19:12
dansmithfungi: yeah I was just hearing about one local company the other day that said "you know, we could more than pay for everyone's internet if we stop rending a box 24 hours a day that only gets used for 8"19:12
dansmith*renting19:12
jungleboyj:-)19:12
jungleboyjI like rending a box as well.19:13
fungirenting it, but then also rending it19:13
fungiit really does end up being both19:13
* jungleboyj laughs19:13
fungibut yes, the costs of maintaining office space are, in many cases, greater than the benefits19:14
fungiwithout a year or two long experiment in not using office buildings, it would be much harder to convince a lot of companies of that fact, but now they have the numbers19:14
dansmithyeah19:14
dansmithso...thank you pandemic? no, that's too far.19:15
dansmithit's done wonders for my antisocial tendencies, I'll say that ;)19:15
jungleboyj:-)  There has been upsides from it.19:15
jungleboyjdansmith: How much more could be done.  ;-)19:15
dansmithheh19:15
fungiwhen i had an hour-long commute each way every day, i never managed to get to work on time. once i started working from home, i was never late!19:15
dansmithnow that my wife WFH too there's a little traffic on my commute, but it's very manageable :)19:16
fungialso, all that time no longer wasted travelling back and forth, costly fuel and vehicle maintenance, et cetera19:16
dansmithyeah19:16
jungleboyjWe have a traffic jam on the stairs once in a while.  Especially if the dog commutes at the same time ... but not too bad.20:32
fungiwe have a cat who is determined to assassinate us and make it look like an stairwell-related accident20:35
jungleboyj:-)  I think our cat tries that once in a while.  She is too lazy in the morning to be a part of the commute.20:36
dansmithfungi: hah. my cat is fine with having blood on his claws following his assassination operation20:36
jungleboyjOur dog is a Dachshund though.  He looks like a bratwurst.  So we joke about having a jack knifed sausage on the stairs blocking traffic.20:36
opendevreviewMerged openstack/governance master: Add slaweq nomination as vice-chair  https://review.opendev.org/c/openstack/governance/+/83317123:17
opendevreviewMerged openstack/governance master: Appoint Wu Chunyang as Trove PTL  https://review.opendev.org/c/openstack/governance/+/83315223:17
opendevreviewGhanshyam proposed openstack/governance master: Reset TC liaisons list  https://review.opendev.org/c/openstack/governance/+/83424323:47

Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!