15:00:25 <gmann> #startmeeting tc
15:00:25 <opendevmeet> Meeting started Thu Mar 17 15:00:25 2022 UTC and is due to finish in 60 minutes.  The chair is gmann. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:00:25 <opendevmeet> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
15:00:25 <opendevmeet> The meeting name has been set to 'tc'
15:00:28 <gmann> #topic Roll call
15:00:30 <gmann> o/
15:00:37 <rosmaita> o/
15:00:43 <diablo_rojo> o/
15:00:51 <dansmith> o/
15:00:59 <slaweq> o/
15:01:08 <jungleboyj> o/
15:01:30 <spotz_> o/
15:02:01 <gmann> #link https://wiki.openstack.org/wiki/Meetings/TechnicalCommittee
15:02:04 <gmann> Today agenda ^^
15:02:13 <gmann> let's start
15:02:16 <gmann> #topic Follow up on past action items
15:02:20 <gmann> no action item form last meeting
15:02:29 <gmann> #topic Gate health check
15:02:41 <gmann> any news on gate?
15:03:21 <dansmith> there was a creeping failure in the nova-ceph-multistore job,
15:03:36 <dansmith> which was OOMing mysql, which I hopefully fixed by trimming down
15:03:42 <knikolla> o/
15:03:45 <dansmith> that affected a few projects at least
15:04:03 <gmann> +1
15:04:14 <dansmith> I know we've still got the volume detach failure thing going on with the centos jobs, at seemingly 100%, but those aren't voting anywhere that I know of
15:04:25 <gmann> yeah, I am able to make rescue server test pass with SSH-able server fix but there are two more test failing #link https://review.opendev.org/c/openstack/tempest/+/831608
15:04:50 <dansmith> oh I thought all the sshable fixes were merged
15:05:06 <dansmith> oh right, I remember this one now, nevermind
15:05:07 <gmann> even after unrescue server we need to wait for ssh-ready before detach happening
15:05:25 <dansmith> this breaks non-centos jobs now right?
15:05:27 <fungi> we upgraded gitea yesterday, and this exposed a regression with pip install of git remotes from it which impacted blazar's jobs because they were still configured to try zuul v2 era zuul-cloner workflows
15:05:29 <gmann> yeah, not all. rescue test was seen as failure in reported bug but there are few more
15:05:50 <gmann> dansmith: no, they will pass. i did recheck.
15:06:07 <dansmith> gmann: oh, then why haven't we merged it yet?
15:06:07 <fungi> i've got a fix in the pipe to address the pip install errors, but also it exposed that their jobs are sorely in need of modernizing
15:06:22 <gmann> dansmith: I was trying a litlte smart for having active server then rescue/unrescue and then SSH but that did not work
15:06:38 <gmann> dansmith: It just passed yesterday might so will merge after gate pass :)
15:06:42 <dansmith> oh okay
15:06:43 <gmann> *yesterday night
15:07:28 <gmann> fungi: you mean their jobs also on zuulv2?
15:08:08 <jungleboyj> On a related note the Cinder team is starting a renewed effort to get people to not do 'naked rechecks' .
15:08:19 <dansmith> oh very nice
15:08:21 <gmann> +1
15:08:25 <rosmaita> \o/
15:08:27 <fungi> gmann: more that their jobs hadn't been touched since the zuul v2 days, so were trying to use the zuul-cloner tool to checkout openstack/nova, and because that's not a thing any longer they were falling back to pip install nova from a git remote
15:08:38 <dansmith> jungleboyj: I try to shame people when I find them doing that with evidence that it's wrong :)
15:08:50 * jungleboyj isn't surprised
15:08:54 <jungleboyj> :-)
15:09:01 <rosmaita> dansmith: we expect no less of you
15:09:07 <dansmith> but it works ;)
15:09:07 <gmann> slaweq had good script to collect recheck numbers.
15:09:11 <slaweq> speaking about rechecks, I prepared some data as we talked last week
15:09:17 <spotz_> *hides*
15:09:20 <gmann> yeah, I was coming to that
15:09:20 <slaweq> https://docs.google.com/spreadsheets/d/1zlJixCttF1e7ZSJdZzORfrJWRqsllRDn97FHyTr11e8/edit?usp=sharing
15:09:30 <slaweq> here are data for "main" repositories
15:09:54 <slaweq> and if You want it for each repo in openstack/ there is tar.gz file http://kaplonski.pl/files/openstack_rechecks_data.tar.gz
15:10:20 <slaweq> I collected data for each repo from openstack from last 365 days
15:10:40 <diablo_rojo> Oh thats a pretty cool visualization.
15:10:49 <gmann> very nice.
15:10:56 <dansmith> I'm not really sure what the numbers are though
15:11:11 <slaweq> the number there are basically average number or rechecks done on last PS before it was merged
15:11:13 <gmann> is it like <2 recheck per week for most project? or I am reading wrongly ?
15:11:13 <dansmith> is this rechecks or just build failures?
15:11:16 <jungleboyj> dansmith:  Ok, good.  Not just me.
15:11:24 <slaweq> average for every week
15:11:25 <dansmith> slaweq: ah okay
15:11:30 <jungleboyj> Ah ...
15:11:43 <dansmith> slaweq: so higher numbers potentially mean recheck grinding to get a patch in?
15:11:52 <slaweq> dansmith: right
15:11:59 <gmann> ohk so it is per patch not all jobs
15:12:13 <slaweq> it's per patch and average per week
15:12:13 <gmann> then 1 or 2 still high number per patch
15:12:15 <jungleboyj> So, on average ever other patch has to go through a recheck to merge.
15:12:24 <gmann> jungleboyj: yeah
15:12:35 <jungleboyj> Wow.
15:12:36 <fungi> is that just for gate pipeline failures, or check as well?
15:12:44 <gmann> good question ^^
15:12:45 <slaweq> fungi: both
15:13:01 <gmann> check might have mostly but yeah gate also have recheck
15:13:08 <fungi> so given that the patch has to pass once in check and once in gate in order to merge
15:13:08 <slaweq> I was basically counting "recheck" comments on last patch sets
15:13:31 <slaweq> of course there may be some patches where rechecks were done "on purpose"
15:13:44 <slaweq> but in general I think it's not very common practice
15:13:57 <jungleboyj> Fair assumption.
15:14:01 <fungi> also note that long patch series and/or depends-on can skew this, since one change failing can cause all the ones which rely on it to also fail
15:14:14 <gmann> slaweq: ohk so on last PS not all recheck on that commit ?
15:14:28 <slaweq> fungi: true, it's not ideal metric for sure
15:14:54 <jungleboyj> Oh wow, if it is just the last patch set, then the actual number of rechecks per patch could be higher.
15:14:56 <slaweq> gmann: yes, I was counting only last patch set as I assumed that if that PS was merged finally, it means it was good
15:15:01 <gmann> yeah, DNM, testing patch also in that but that is ok
15:15:21 <slaweq> DNM patches aren't in that metric. I was filtering only merged patches
15:15:29 <gmann> k, +1
15:15:42 <fungi> jungleboyj: but remember that this includes check, so includes the "surely the problem isn't my patch, oh i guess maybe it is?" rechecks too
15:16:02 <fungi> or would if you included patch sets before the final one
15:16:09 <slaweq> I'm using this script and metric in neutron since some time and even if it's not great it shows us pretty clearly current state of the Neutron CI :)
15:16:25 <jungleboyj> ++
15:16:32 <jungleboyj> Some data is better than no data.
15:16:36 <dansmith> yeah I think this probably gives us a good view of how much rechecking needs to happen to get something to land,
15:16:42 <slaweq> https://github.com/slawqo/tools/blob/master/rechecks/rechecks.py
15:16:46 <slaweq> that is script
15:16:48 <gmann> slaweq: and is it lot of data if we collect for all recheck including previous recheck that merge PS?
15:16:59 <dansmith> but probably needs a bit more to tell us more than that, like if individual patches are actually rechecked more than the average, etc
15:17:00 <gmann> dansmith: yeah
15:17:10 <dansmith> but as a heartbeat sort of thing, if the graph goes up -> bad
15:18:19 <slaweq> dansmith: I can prepare some data "per patch" too
15:18:33 <slaweq> I will need to modify that script but it shouldn't be hard
15:18:40 <gmann> yeah and we can like ignore patches if just 1 recheck or so if data is more
15:19:05 <dansmith> slaweq: I'm not asking you to do that, just suggesting, but yeah always nice to have data and more data :)
15:19:05 <gmann> slaweq: just to make sure it is just 'recheck' not recheck with reason?
15:19:24 <gmann> or both
15:19:40 <slaweq> gmann: actually it is counting number of comments like "Build failed" on the last PS
15:19:49 <slaweq> https://github.com/slawqo/tools/blob/master/rechecks/rechecks.py#L155
15:19:53 <dansmith> I really hate that we've drifted away from "recheck with reason" .. I wish we could encourage better behavior there sometime
15:19:56 <dansmith> *somehow
15:20:22 <diablo_rojo> The same way you should be descriptive with your commit message, you should be descriptive with why you are rechecking.
15:20:38 <dansmith> diablo_rojo: I agree, and I never do naked rechecks, but I'm in the minority
15:20:45 <fungi> "recheck with reason" has always been optional. people who are inclined to look into and record why they're rechecking something will do it regardless of whether it's expected, and people who don't want to bother will make something up like they did back when we enforced it
15:20:53 <diablo_rojo> I know I am not innocent when it comes to rechecks without explicitly saying why.
15:20:53 <slaweq> dansmith: sure, I will do modification to have data "per patch"
15:20:55 <spotz_> I just kind of assumed it triggered off the commit just being recheck
15:20:56 <dansmith> and I was shamed for my use of shame, so.. lollipops? :)
15:21:04 <slaweq> I don't know if for next week but I will do that
15:21:17 * jungleboyj is guilty as well.
15:21:19 <dansmith> fungi: yeah I know
15:21:24 <spotz_> My guess is training issue. We see just recheck so we use just resheck
15:21:33 <diablo_rojo> Sure its optional, but it would be better if we made it the majority rather than a minority.
15:21:39 <gmann> slaweq: no hurry. I am going to add this recheck script/data in PTG and we can discuss what all data we want to monitor per week in zed
15:21:42 <diablo_rojo> Wouldn't want dansmith feeling lonely, you know.
15:21:54 * dansmith sobs uncontrollably
15:22:03 <gmann> easy data will be per week as we monitor weekly so will be easy to check even for all patches or per patches
15:22:05 <slaweq> gmann: sure, great idea. I will be more than happy to help with that
15:22:06 <dansmith> spotz_: it used to be required, but people would just "recheck foo" or "recheck bug 00000"
15:22:07 <jungleboyj> There there ...
15:22:11 <fungi> i agree it's a good practice, but it's not a good source of data unfortunately because of the number of people who knowingly pollute it
15:22:35 * diablo_rojo hands dansmith a handkerchief "there there"
15:22:44 <gmann> slaweq: thanks for this.
15:22:45 <dansmith> maybe we should try to encourage PTLs to push the better behavior in their teams
15:23:01 <jungleboyj> dansmith:  That is where Cinder is starting.
15:23:06 <diablo_rojo> slaweq, yes thanks for the data. I look forward to the per patch info!
15:23:13 <gmann> sure, how? in TC+PTL sessions or in ML?
15:23:17 <dansmith> jungleboyj: ack, well, let's try to spread that
15:23:25 <dansmith> gmann: yeah we could start in the PTG session
15:23:28 <gmann> +1
15:23:37 <gmann> I will add it
15:23:44 <slaweq> yeah, in neutron we are trying to do "recheck with reason" too but it's not always easy
15:23:46 <dansmith> gmann: cool
15:23:50 <jungleboyj> +2
15:23:56 <slaweq> and I also don't do it sometimes :/
15:24:04 <slaweq> but I will try to do better :)
15:24:12 <gmann> #action gmann to add recheck data topic in PTG etherpad (TC and TC+PTL for awareness)
15:24:12 <dansmith> slaweq: be the change.. be the change.. :P
15:24:22 <gmann> +100
15:24:32 <rosmaita> this is our advice in cinder: https://docs.openstack.org/cinder/latest/contributor/gerrit.html#ci-job-rechecks
15:24:33 <slaweq> dansmith: yes sir! :D
15:24:44 <dansmith> :)
15:24:48 <spotz_> heheh
15:24:51 <dansmith> 24 minutes in and still on gate, eh?
15:24:54 <rosmaita> just putting it out there, because i don't know that we are generating machine parseable comments
15:25:42 <jungleboyj> It is our favorite topic dansmith
15:25:42 <dansmith> rosmaita: "recheck I don't know but at least I looked" is better to me than nothing
15:25:57 <gmann> fungi: coming back to blazer issue, do you have link for that/job or know if they are working to fix their side?
15:26:54 <fungi> priteau is working on it, but it was jobs for blazar-nova specifically
15:26:57 <gmann> rosmaita: I see them sometime machine generated comment (not recheck) and that annoy me more than anything
15:27:08 <gmann> fungi: ok.
15:27:42 <rosmaita> i have tried to get out third-party ci to add the appropriate gerrit tag so they don't pollute the comments, but you can see how much success i have had
15:27:55 <gmann> and as frickler pointed out today I pushed moving l-c job to focal/py38 but there are existing config error in that field which needs to be fixed
15:28:45 <fungi> gmann: an old tools/tox_install.sh in blazar-nova specifically
15:28:57 <gmann> fungi: I see
15:29:13 <gmann> good discussion on gate things today. anything else?
15:29:19 <dansmith> +1000
15:29:40 <dansmith> very glad to see the gate getting proper attention
15:29:48 <rosmaita> slaweq: nice work on that script, btw
15:29:51 <gmann> true, +10000 :)
15:29:55 <slaweq> thx
15:30:04 <dansmith> gmann: you just had to +10x me huh?
15:30:11 <gmann> yeah
15:30:13 <dansmith> hah
15:30:29 <gmann> #topic Z cycle Leaderless projects
15:31:08 <gmann> only 1 project adjutant left which we are waiting until March end. we will discuss that in PTG
15:31:14 <gmann> I will remove it from genda
15:31:23 <gmann> #topic PTG Preparation
15:31:40 <jungleboyj> ++
15:31:42 <gmann> #link https://etherpad.opendev.org/p/tc-yoga-ptg
15:31:51 <gmann> #link https://etherpad.opendev.org/p/tc-ptl-interaction-zed
15:31:58 <gmann> please add topic in those etherpad
15:32:14 <gmann> timeslots are finalized and I have updatad it on ML as well as in etherpad
15:32:44 <fungi> note that the schedule and precreated etherpad links are now live in ptgbot, so can probably safely start adding overrides if needed. diablo_rojo would know for sure though
15:32:52 <knikolla> the first one is the link from the yoga ptg :)
15:32:58 <gmann> I have informed Kubernets steering committee for joining us in PTG
15:33:00 <spotz_> gmann do we need to bring up Sahara, Magnum, etc there or will it be too late
15:33:10 <diablo_rojo> I think you can override it now.
15:33:29 <gmann> sorry #link https://etherpad.opendev.org/p/tc-zed-ptg
15:33:40 <jungleboyj> here we go.
15:33:44 <gmann> spotz_: in TC+PTL sessions?
15:33:49 <spotz_> Yeah
15:34:11 <gmann> yeah, we call everyone actually not specific projects
15:34:13 <diablo_rojo> Everything looks correct at this point. I just need to do the zoom room setup once we are closer but that shouldn't affect other things.
15:34:30 <gmann> diablo_rojo: +1, nice
15:35:29 <gmann> spotz_: and for less active/broken project like sahara, magnum we can address/ping them separately. I would like to keep TC+PTL sessions to get/give feedback sessions instead of going towards project health checks
15:36:00 <gmann> if we do project health check many PTLs will not join :)
15:36:20 <spotz_> gmann ok
15:36:40 <gmann> spotz_: for magnum I know there are few new cores in last cycle which you can ping.
15:36:41 <jungleboyj> Probably true.
15:37:23 <gmann> anything else on PTG?
15:37:57 <diablo_rojo> Please register if you havent yet!
15:38:15 <gmann> +1, i did.
15:38:28 <slaweq> Me too 🙂
15:38:40 <gmann> #topic Open Reviews
15:38:43 <gmann> #link https://review.opendev.org/q/projects:openstack/governance+is:open
15:38:45 <diablo_rojo> \o/
15:39:01 <gmann> I need one more vote on slaweq vice-chair nomination #link https://review.opendev.org/c/openstack/governance/+/833171
15:39:15 <spotz_> voted:)
15:39:20 <gmann> all other open reviews are good, either waiting for time or PTL +1
15:39:36 <gmann> thanks, that is all from my side today. anything else to discuss?
15:39:41 <diablo_rojo> nice :)
15:39:45 <gmann> we have around 21 min
15:40:03 <jungleboyj> Voted!  Thank you slaweq !
15:40:18 <gmann> thanks and yes thanks slaweq for volunteer
15:40:26 <spotz_> Assuming we have joint leadership in Berlin do we want to do anything separate from that?
15:40:37 <slaweq> yw, I hope I will learn quickly and be able to help gmann there :)
15:40:45 <gmann> slaweq: +100
15:41:05 <diablo_rojo> Forum submissions should be opening next week I think
15:41:20 <gmann> spotz_: I think that is good one to restart. and joint leadership meeting is enough at least for Board interaction
15:41:56 <spotz_> Sounds good, I pinged the OPS Meetup folks as we're 10 weeks out and really need to get planning
15:42:00 <gmann> diablo_rojo: on Forum sessions, do we need TC volunteer for selection committee like we used to have?
15:42:12 <gmann> spotz_: +1 on ops meetup.
15:42:17 <diablo_rojo> I have a few PTL volunteers actually
15:42:28 <diablo_rojo> So we are good for OpenStack forum selection representation
15:42:31 <spotz_> I told her I would if no one else staeeped up
15:42:46 <diablo_rojo> That too :)
15:43:21 <gmann> dansmith: nice, I saw wiki and if i understand correctly requirement is not two TC has to be in selection but it can be anyone from community right?
15:43:33 <gmann> diablo_rojo: ^^
15:43:37 <diablo_rojo> lol
15:43:40 <gmann> dansmith: please ignore
15:43:46 <dansmith> aheh
15:43:50 <dansmith> I was like ..uhh
15:43:50 <gmann> your both name with d* :)
15:43:53 <diablo_rojo> Yeah it can be anyone from the community just ideally someone in a governance position
15:44:03 <diablo_rojo> so PTLs are great too
15:44:50 <diablo_rojo> Yeah dansmith, here I thought you were the Forum expert lol
15:44:52 <gmann> "1 delegate from each OpenInfra Project
15:44:52 <gmann> 2 OpenInfra Foundation staff members"
15:45:02 <diablo_rojo> yep
15:45:13 <gmann> diablo_rojo: may be good to mention that clearly about governance in that
15:45:28 <diablo_rojo> gmann, it says elsewhere in the wiki I believe
15:45:34 <gmann> #link https://wiki.openstack.org/wiki/Forum
15:46:09 <gmann> "The TC and UC are best placed to decide which of their members should represent each body...." may be this line can be modified now?
15:46:20 <gmann> this is left om previous requirement?
15:46:29 <diablo_rojo> Ah yeah that needs to be updated.
15:46:34 <diablo_rojo> I will tweak later today
15:46:42 <gmann> k, just making sure we do not miss anything from TC which we need to do
15:46:47 <gmann> diablo_rojo: thanks
15:46:53 <diablo_rojo> I would let you know if we were :)
15:47:01 <gmann> great.
15:47:15 <gmann> anything other topic to discuss/
15:48:12 <gmann> #endmeeting