17:59:04 <knikolla> #startmeeting tc
17:59:04 <opendevmeet> Meeting started Tue Jul 25 17:59:04 2023 UTC and is due to finish in 60 minutes.  The chair is knikolla. Information about MeetBot at http://wiki.debian.org/MeetBot.
17:59:04 <opendevmeet> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
17:59:04 <opendevmeet> The meeting name has been set to 'tc'
17:59:21 <knikolla> Hi all, welcome to the weekly meeting of the OpenStack Technical Committee
17:59:25 <knikolla> A reminder that this meeting is held under the OpenInfra Code of Conduct available at https://openinfra.dev/legal/code-of-conduct
17:59:29 <knikolla> Today's meeting agenda can be found at https://wiki.openstack.org/wiki/Meetings/TechnicalCommittee
17:59:33 <knikolla> #topic Roll Call
17:59:35 <spotz[m]> o/
17:59:36 <rosmaita> knikolla: you are in a hurry to get started!
17:59:37 <noonedeadpunk> o/
17:59:37 <jamespage> o/
17:59:43 <rosmaita> o/
18:00:03 <knikolla> o/
18:00:07 <dansmith> o/
18:00:08 <gmann> o/
18:00:33 <knikolla> rosmaita: haha, a bit, yes.
18:00:42 <JayF> o/
18:00:58 <knikolla> No noted absences for today.
18:01:08 <slaweq> o/
18:01:31 <knikolla> and everyone's already here, nice.
18:01:43 <knikolla> #topic Follow up on past action items
18:01:51 <knikolla> knikolla will amend proposal for Unmaintained branches.
18:02:09 <knikolla> I have, we can talk more about it during its respective topic.
18:02:14 <knikolla> tc-members to review https://review.opendev.org/c/openstack/project-team-guide/+/843457
18:02:35 <knikolla> I haven't had the time, I hope others had a chance to take a look.
18:03:01 <rosmaita> got some good comments from gmann
18:03:15 <gmann> the idea lgtm but we need to see the automation of it otherwise it requires extra manual work which might endup project going in just link/copy the previous release notes
18:03:16 <knikolla> Awesome. We can go in more detail during its respective item as well.
18:03:17 <knikolla> Moving on.
18:03:28 <knikolla> #topic Extended Maintenance
18:03:34 <noonedeadpunk> yeah ++ for some automation concept at least on paper
18:03:34 <knikolla> #link https://review.opendev.org/c/openstack/governance/+/888771
18:04:00 <knikolla> Got some really good discussion on the Unmaintained governance patch and I made revisions to bring it aligned to our previous discussions and suggestions.
18:04:18 <knikolla> Please take a look and let me know if you have any questions or things you'd like to bring up related to the topic.
18:04:23 <noonedeadpunk> I haven't checked since yesterday update :(
18:04:35 <gmann> thanks, latest version lgtm
18:04:52 <dansmith> there are still some things to clarify there
18:05:04 <dansmith> but those aside I'm good with what's there now
18:05:37 <rosmaita> i'm still a bit unclear about whether the unmaintained branch is created automatically, or is opt-in
18:05:44 <knikolla> Especially related to the processes of opt-in and opt-out. We just mention that there'll be a document to refer to.
18:06:05 <gmann> for current EM, it is 3 automatic opn-in
18:06:11 <knikolla> Creation is automatic. But a team can opt-out after it's created similar to the EOL process right now.
18:06:22 <gmann> but that seems not alligned with what rosmaita mentioning of cinder team plan
18:06:28 <knikolla> (at least that's what I imagine, obviously open to feedback)
18:06:31 <JayF> I really liked the proposal, as written, where the branch gets created by default the first time then retired if not renewed.
18:06:51 <JayF> For teams like Cinder, can't they just ignore the unmaintained/ branch, just as we were hoping folks would for EM if nobody came around to help?
18:07:09 <gmann> so no automatic creation for cinder ?
18:07:10 <rosmaita> well, why create it in the first place if no one is going to use it?
18:07:17 <fungi> seems like the end-of-month deadline for getting a recommendation back to the cinder team is rapidly approaching (august starts one week from today), just as a time-check
18:07:18 <knikolla> I also think that with the rename to Unmaintained, there should be significantly less reasons for a team to opt-out, but more likely just to not to renew.
18:07:38 <JayF> If we haven't solved that basic ill: a project team feels too strong of ownership to leave a branch unmaintained, then we still have a problem, yeah?
18:07:41 <knikolla> So I would hope that Cinder reconsiders.
18:07:48 <JayF> rosmaita: let me tell you why I like it: because we demonstrate that those people don't exist
18:07:55 <gmann> knikolla: ++, that is what I understood. it will a good signal on hey we had unmaintiand brnahces created and no one show up to shutting down
18:08:04 <JayF> rosmaita: there is value in having an actual year-old stale branch to point at when people complain about you shutting things down
18:08:09 <slaweq> rosmaita how can we know from the beginning that nobody is going to use it? Maybe people will step in during e.g. first cycle when branch is unmaintained?
18:08:19 <slaweq> IMO we should give them a chance for that
18:08:27 <gmann> slaweq: yeah, that also possible
18:08:32 <rosmaita> well, i am judging by the silence on the ML about the current cinder EOL proposal
18:08:34 <fungi> conversely though, why create it until someone expresses an interest in having it?
18:08:44 <dansmith> yeah, and also it helps us avoid the situation where someone wants to re-open an EOL'd branch
18:08:51 <gmann> and when will see there are unmaintained brahcnes for nova and they need cinder too so they might show up to maintain
18:09:23 <knikolla> fungi: i think it lowers the barrier for someone that has an interest.
18:09:37 <knikolla> but doesn't put undue burden on the team itself.
18:09:53 <JayF> rosmaita: bluntly, I mostly agree with you here and would rather a dead branch not be created in the first place; but I think we have to demontrate the dire-ness of the circumstances around those old branches to get the mindshare to change further
18:09:55 <gmann> rosmaita: and it will not ask project team to maintain it, it is just similar to what cinder team asking in ML. a better communication of asking maintianence who need it
18:10:22 <gmann> JayF: we do not know if that will be dead or not right? by creating those we will get to know fr sure
18:10:36 <JayF> gmann: If someone wanted that branch to be alive, they've had months to step up at this point
18:10:50 <dansmith> JayF: we're talking about future branches though right?
18:10:52 <JayF> gmann: I am convinced those contributors don't exist; but I'm also convinced that words in IRC aren't likely to change minds here
18:10:58 <gmann> JayF: ML is not always perfct place to communicate what we want. it is just a one of the way
18:11:04 <JayF> dansmith: I am using the past to predict future events :)
18:11:10 <rosmaita> ok, so how is this going to work for cinder right now?  we are proposing EOL of train, ussuri, victoria, wallaby, and xena
18:11:27 <rosmaita> i guess train and ussuri are fine to go EOL & be deleted
18:11:34 <gmann> my point is Cinder communicated it on ML and let's do the same via new process of unmaintined branch creation
18:11:45 <noonedeadpunk> I actually +1 to Seans proposal and consider Y applicable for unmaintained
18:11:48 <gmann> rosmaita: yes, only Victoria, wallaby, xena for now
18:12:11 <noonedeadpunk> To be same "testbed" as for SLURP
18:12:14 <spotz[m]> And Yoga?
18:12:14 <rosmaita> so, v, w, and x will go to 'unmaintained' along with everyone else?
18:12:42 <knikolla> If Cinder EOL their branches before we cut the unmaintained branches, no branches will be created for them, based on current language of the proposal.
18:12:57 <gmann> rosmaita: only v,w,x will be unmaintined automatically and rest all not
18:13:03 <dansmith> JayF: I see 7 pending backports from people not associated with the core team against nova's stable/wallaby right now.. so let's not say they completely don't exist
18:13:19 <noonedeadpunk> based on current language only SLURP will have unmaintained, which starts from Antelope
18:13:28 <JayF> dansmith: I'm speaking for the Cinder case
18:13:45 <knikolla> noonedeadpunk: I added a new section in the proposal in the bottom about current branches.
18:13:52 <gmann> rosmaita: asper current proposal we will create unmaintained/victoria , unmaintained/wllaby, and unmaintained/xena as automatic opt-in and test all are not not automatic opt-in
18:13:52 <JayF> dansmith: Part of why there's value in treating them separate in policy; Cinder has fewer resources than Nova generally, so supporting branches is a larger burden
18:14:06 <gmann> s/test/rest
18:14:07 <dansmith> JayF: I don't think they do lately, fwiw
18:14:22 <gmann> JayF: cinder team do nto need to support
18:14:32 <dansmith> cinder looks to me to have a number of similar backports proposed to wallaby
18:14:51 <dansmith> just saying, we're trying to clear the air here and open up some space to make it easier for more of those people to be here,
18:14:57 <gmann> it is just opening unmaintained branches and project team will not do any work there. if no one shows up then unmaintianed branches move to EOL
18:15:08 <dansmith> so we shouldn't say "well it was super hard and obscure how to do this in the past, so those people must not exist"
18:15:45 <JayF> dansmith: that's why I have been saying, as a demo, I'm onboard for opening the first year universally and seeing what happens :) I don't think anything will happen but I've been wrong in the past will be again :D
18:16:22 <dansmith> mkay, seems like you're saying the opposite, but ... okay
18:16:23 <noonedeadpunk> knikolla: ah. sorry, haven't finished yet...
18:16:58 <gmann> that is what we are tryting with this new model, instead of shutdown EM concept, let's open up the things one more time but this time with people gets what they signup to maintain it
18:17:16 <JayF> dansmith: I'm saying I empathize a lot with rosmaita's position and could easily hold it if the majority was there :) I do hold that if a bunch of people disagree it's a sign that maybe more caution is required :D
18:18:17 <gmann> rosmaita: what extra work you think on cinder team with these 3 unmaitained branches ?
18:18:36 <gmann> after initial setup
18:19:02 <knikolla> well, the PTL has to assign an unmaintained liaison. that would be on the Cinder team.
18:19:08 <rosmaita> i'm just trying to understand the proposal and how these branches will be killed off eventually
18:19:25 <gmann> knikolla: just liaison but nto maintenance right?
18:19:28 <JayF> knikolla: so lets say, in Cinder's case, there is nobody vvolunteering to be a liason
18:19:44 <JayF> knikolla: what happens then? Because essentially Cinder saying "we only want to care about supported branches" is implying that's extremely likely
18:19:59 <gmann> it is just a liason to keep track of unmaintained branch not to MAINTAINED them
18:20:01 <knikolla> JayF: in that case the PTL has the Unmaintained liaison role, but nobody is interested in Unmaintained so no action item for the PTL at all.
18:20:21 <knikolla> Branches get created and killed off automatically with no intervention
18:20:23 <JayF> so PTL at least has to be traffic cop for unmaintained/
18:20:33 <JayF> for a year, ushering anyone interested into the core group
18:20:43 <JayF> if that's the case, I'd want extremely clear guidelines saying "give access"
18:21:02 <noonedeadpunk> to be frank, populating unmaintained group manually sounds like a pita for small projects
18:21:17 <JayF> I do not want to be in a situation, as Ironic PTL, where the community wants something retired but Shade E. Character comes along and says "I'll take over this unmaintained branch"
18:21:20 <knikolla> PTL assigns Unmaintained Liaison. Unmaintained Liaison manages the ACLs for the unmaintained-core group.
18:21:27 <gmann> noonedeadpunk: it is just 1 branch with SLURP (3 for now pre-SLURP)
18:21:29 <noonedeadpunk> I would add some default there and then anyone who wants on top
18:21:44 <noonedeadpunk> gmann: we have plenty of projects just with 2-3 maintainers
18:21:53 <noonedeadpunk> (at best)
18:22:07 <JayF> we have plenty of projects where only 2-3 maintainers would do that kind of maintanance :)
18:22:10 <knikolla> Would it be helpful to have a TC managed unmaintained-core across all project Unmaintained branches, for known good actor operators?
18:22:16 <JayF> knikolla: yes
18:22:18 <noonedeadpunk> so saying them that to unmaintained they need to populate the group each time is kinda meh...
18:22:18 <gmann> noonedeadpunk: yes but there is zero maintained work on unmaintained branches from project team.
18:22:34 <JayF> knikolla: As PTL of $project, giving me decision power over unmaintained branches means that I'm still responsible for them
18:22:35 <gmann> why we think adding someone in core group is lot of work
18:22:44 <JayF> gmann: it's not work-work, it's responsibility-work
18:22:45 <noonedeadpunk> but why to prohibit maint team by default to vote if the want to?
18:22:53 <noonedeadpunk> *if they want to
18:23:00 <gmann> JayF: which is 30 sec in a 6 month
18:23:02 <JayF> gmann: if I give access to ironic-unmaintained-core to someone who abuses it, that's going to feel like I screwed up
18:23:09 <knikolla> noonedeadpunk: because if they can they will, through a sense of responsibility.
18:23:14 <noonedeadpunk> like reviewing some backport might be easy enough if they have permissions, but for PTL allowing that is pita
18:23:21 <JayF> gmann: that's what I mean, not that it's work to click the buttons; it's explicitly giving responsibility that PTLs are trying to disavow
18:23:26 <gmann> JayF: that can heppen in current core also what we do in that case?
18:23:38 <JayF> gmann: PTLs signed up to maintain core lists for maintained branches :)
18:23:46 <noonedeadpunk> knikolla: sense of responsibility somehow is not applicable to PTLs?
18:23:51 <knikolla> if the PTL wants, they can include the entire stable-maint group.
18:23:56 <gmann> yeah, so what is extra respo in that
18:24:04 <JayF> this is all about trying to ensure the existing maintenance group can disavow themselves of a unmaintained/ branch
18:24:16 <JayF> if that group is the keyholder to that unmaintained/ branch, then they are responsible
18:24:21 <knikolla> Some projects already include <project>-core in <project>-stable-maint.
18:24:25 <gmann> PTL do this task for any branch core group in that project
18:24:50 <gmann> let's do like this
18:24:51 <noonedeadpunk> I really don't get that point of repsonsibility
18:25:24 <slaweq> JayF IMO we are going a bit too much into what can possibly go wrong, currently for many smaller projects we, as TC are assigning PTL after election just because someone raise hand and said that will do it
18:25:25 <JayF> noonedeadpunk: I'm having a hard time explaining it because it's something that's really core to me personally, so I've never had to tease it out. If you give me the keys to something, and I open it for them, I'm responsible for what they do with that access.
18:25:30 <gmann> 1. assign  stable-maint group (core stable group nit project stable group) to unmaintained branches as default
18:25:30 <gmann> 2. they can add new people if there is any
18:25:35 <gmann> 3. n work on PTL
18:25:38 <gmann> no work
18:25:44 <slaweq> in such case that person can not only screw unmaintained branches but also master
18:26:47 <fungi> distilling the burden of responsibility into the tasks which go along with that responsibility is missing a large chunk of what responsibility means. even being responsible for something which has no accompanying tasks can still represent a burden on someone (being responsible means you're held accountable for all related decisions and indecisions). or maybe people mean something other
18:26:48 <fungi> than accountability when they say "responsible"
18:26:54 <noonedeadpunk> JayF: well, all projects are community driven. Having no personal responsibility is basically header of each file in each repo under apache 2.0
18:27:03 <knikolla> gmann: i want to avoid having -stable-maint be included by default because there are already people who look at the group of who has approval powers on a branch and send an email to all of them.
18:27:05 <JayF> slaweq: everything you said I agree with; but I come to a different conclusion (mainly I wish we appointed less PTLs, but I can't fix that :D)
18:27:16 <gmann> IMO, PTL/liaison keep track of unmaintained branch and add people is just a very small resp and work.
18:27:21 <JayF> fungi: you nailed my concern, on the button
18:27:42 <slaweq> gmann++
18:27:45 <gmann> knikolla: yeah, that is true
18:27:54 <noonedeadpunk> fungi: do you know if PTL can jsut add group to group rather then list everyone explicitly?
18:27:59 <JayF> fungi: and almost definitionally, if these folks are outside of the existing community, $PTL has little/no context to know if they are good or bad actor
18:28:07 <noonedeadpunk> As I can't recall that being possible
18:28:12 <knikolla> i'm pretty sure they can, cause we have already done that in keystone.
18:28:24 <noonedeadpunk> or well, at least without pushing a patch to project-config
18:28:30 <JayF> I'm 99% sure that's possible in the web ui
18:28:32 <gmann> let's do this and if we see PTLcannot do this then we can discuss. but this is very small part of the proposal solving many thjings
18:28:40 <noonedeadpunk> ok, gotcha
18:28:46 <JayF> I think this is the nexus of the change though, right?
18:28:49 <knikolla> https://review.opendev.org/admin/groups/ed7e98cd14f0ec4dfd09d10bbb64e1bd3ac1fa15,members
18:28:55 <fungi> noonedeadpunk: yes, groups can include other groups in gerrit. an owner of a group can set another group as included (i'd need to test whether they can set a group as included without being a member of that group though, if that's important to the question)
18:29:01 <JayF> The core problem: project teams feel ownership and responsibility over branches they shouldn't have ownership and responsibility for
18:29:14 <JayF> we can't punt the question of responsibility when that's one of the big fulcrums this whole problem rests upon
18:29:23 <noonedeadpunk> noonedeadpunk: ah, yes, I see that now
18:29:24 <rosmaita> so as far as branches go, is this what we are agreeing to?   https://etherpad.opendev.org/p/24gf87QcmV6xF4StbLQx
18:29:50 <fungi> noonedeadpunk: but also gerrit acls can inherit from other acls, so things for unmaintained branches could be delegated to a central acl through inheritance
18:30:15 <JayF> honestly, a centralized, instead of per-project, UM core group could make sense
18:30:19 <JayF> have TC or a TC liason manage it
18:30:21 <fungi> we already have openstack project acls inheriting from a general openstack acl that adds release manager perms
18:30:31 <noonedeadpunk> rosmaita: and I guess we delete Stein/Train now?
18:30:32 <JayF> you can safely assume operators would be running >1 openstack service, and might have interest in >1
18:30:50 <rosmaita> noonedeadpunk: well, i'm just talking about cinder
18:30:56 <noonedeadpunk> aha, ok
18:31:08 <gmann> rosmaita: nice, all good except when 2023.1 is unmaintined https://etherpad.opendev.org/p/24gf87QcmV6xF4StbLQx#L19
18:31:16 <knikolla> rosmaita: pretty much correct with the exception of, when 2023.1 becomes unmaintained, all other branches are removed and need to be opted-in.
18:31:18 <noonedeadpunk> But we need to have plan for other projects as well:)
18:31:44 <rosmaita> knikolla: ok, please go ahead and correct the etherpad
18:32:08 <knikolla> updated.
18:32:20 <rosmaita> thanks, i somehow missed that in the proposal
18:32:23 <gmann> rosmaita: once we have 2023.1 goes to unmainited it is super easy to have only 1 unmaintained
18:33:09 <knikolla> ++, updated the etherpad to reflect that.
18:33:18 <rosmaita> thanks
18:34:54 <noonedeadpunk> so, we agreed to EOL S/T/U in 2 weeks from now I assume?
18:35:10 <rosmaita> just want to verify that I'm correct that keeping stein/train/ussuri are optional, but x/y/z is required?
18:35:14 <knikolla> rosmaita: do you feel that will make the cinder team willing to keep xwv around?
18:35:24 <JayF> So for LN 23-25, are those opt-in too? or do they have to be forced to eol those?
18:35:47 <knikolla> rosmaita: i removed language that said keeping anything around is required.
18:35:58 <gmann> manual opt-in is always option, we will not force to delete them if anyone interested to maintain he,
18:36:00 <gmann> them
18:36:09 <knikolla> manual opt-out*
18:36:10 <JayF> ack
18:36:12 <JayF> ty gmann
18:37:07 <rosmaita> well, now that we are more clear about what's expected in the 'unmaintained' branches, i don't think there will be a problem keeping v/w/x around
18:37:15 <rosmaita> (as far as cinder is concerned)
18:37:22 <gmann> rosmaita: cool
18:38:03 <rosmaita> ok, i think i am more clear
18:38:18 <knikolla> rosmaita: awesome :)
18:38:55 <rosmaita> next question is do we need to tag for example stable/victoria before renaming the branch to unmaintained/victoria?
18:38:57 <knikolla> and we're representatives of the community. If this process doesn't seem to work for the community either, we can go back to the drawing board.
18:39:51 <spotz[m]> =1
18:39:53 <knikolla> rosmaita: yes, that process we need to document in here https://docs.openstack.org/project-team-guide/stable-branches.html
18:40:12 <knikolla> or have a pointer there to a new document.
18:40:17 <knikolla> as that still says stable branches.
18:40:46 <rosmaita> actually, forget that, we already have victoria-em tag
18:40:56 <gmann> I think new document will be clear and explicit about what is 'unmaintinaed' and it process
18:40:58 <rosmaita> i guess we can create unmaintained/victoria from that?
18:41:12 <noonedeadpunk> yes, makes sense
18:41:18 <noonedeadpunk> everyone have these as EM
18:41:26 <noonedeadpunk> *tagged as EM
18:41:30 <gmann> rosmaita: if there are backport after -em then we need to take that in unmantinaed right ?
18:41:47 <rosmaita> right
18:42:04 <gmann> I think creating new tag and from there create unmaintianed will be good
18:42:05 <noonedeadpunk> wait.
18:42:18 <noonedeadpunk> why not to create unmaintained from top of current stable?
18:42:20 <gmann> and close all the open backport and ask to do the same in new unmaintained brahc
18:42:25 <knikolla> rosmaita: After the meeting can you respond to the ML thread about Cinder EOL with regards to the outcome of this conversation?
18:42:34 <gmann> noonedeadpunk: yes, that is what we need to do
18:42:51 <rosmaita> knikolla: i will do that
18:42:59 <gmann> we can do with latest hash or create new tag -eom like dansmith mentioned
18:43:10 <slaweq> I was just going to ask the same - why You want to ask people to do backports from stable/victoria to u/victoria now is they were merged into s/victoria after em tag?
18:43:12 <dansmith> that was not my idea :)
18:43:24 <noonedeadpunk> it's too late to reject that :D
18:43:25 <gmann> ohk :) I just saw your comment
18:43:57 <gmann> I am ok with tag or not tag but we should create unmaintianed from top of that we have currently on stable/vicrotia
18:44:01 <knikolla> I would like to move on to the next topic item
18:44:04 <noonedeadpunk> ++
18:44:19 <knikolla> this is an implementation detail that doesn't require taking time from the meeting :)
18:44:28 <knikolla> #topic Release notes guidelines for SLURP/NON-SLURP cadence
18:44:32 <noonedeadpunk> what we do with TripleO? :D
18:44:40 <knikolla> #link https://review.opendev.org/c/openstack/project-team-guide/+/843457
18:45:09 <knikolla> noonedeadpunk: ugh...
18:45:11 <gmann> noonedeadpunk: same. wallaby is only the one they want and that goes to unmaintained
18:45:22 <dansmith> yup
18:45:23 <gmann> there they can maintain it
18:46:07 <noonedeadpunk> I'm pretty much fine with that. I expect RedHat not being that fine
18:46:35 <noonedeadpunk> As they exlicitly say it's "maintained"
18:46:47 <dansmith> knikolla: I have not re-reviewed that proposal since last year because I've been busy this week putting out gate fires, which I hope we're going to get to on the agenda
18:46:53 <dansmith> since it's much more pressing than any of this, IMHO
18:47:01 <fungi> well, it was not "maintained" according to openstack's stable branch terminology for quite some time
18:47:09 <knikolla> we can move to the gate then.
18:47:16 <knikolla> #topic Gate Health Check
18:47:17 <gmann> for releasenote
18:47:26 <knikolla> dansmith: floor is all yours
18:47:30 <dansmith> so the gate is very unhealthy at the moment.
18:47:40 <dansmith> gmann and I have been working on timeout issue mitigation all week,
18:47:50 <dansmith> but have been unable to merge most of those patches due to other failures
18:48:18 <dansmith> it seems like we've got a number of neutron-related issues cropping up.. I see a bunch of unreachable network issues and some failed tests about overlapping subnets and things
18:48:32 <dansmith> so we could really use some neutron attention to those things I think
18:48:54 <gmann> I am going to check for more neutron tests if we can speed up those too especially scenario tests
18:48:57 <dansmith> there are also still some cinder-related issues that are causing plenty of troubles, but I'm worried that they're related to IO performance on the workers
18:49:03 <slaweq> dansmith please send me links to those issues, I will try to check them
18:49:09 <dansmith> but if anything has changed recently in cinder that could explain some of that, that'd be good to know
18:49:31 <dansmith> slaweq: ack, I meant to have some links ready for you today, but failed to collect them for $reasons
18:50:05 <rosmaita> i dont' think there have been any major changes to cinder or os-brick that would account for a change like that
18:50:08 <dansmith> slaweq: here's one: https://73af546ea11737b343c2-b256b951e18c7d63775be8a7f99fd99d.ssl.cf5.rackcdn.com/889196/2/gate/tempest-full-py3/9c0d56d/testr_results.html
18:50:09 <gmann> slaweq: this is one of them test_port_security_macspoofing_port , i will send link in neutron channel, yatin was looking into that initially
18:50:10 <slaweq> sure, I will try to take a look this week
18:50:42 <dansmith> slaweq: that link is a volume test failing because of a router issue, so unrelated to actual volumes
18:50:58 <dansmith> and another actual network test that failed, which could be a knock-on issue
18:51:08 <slaweq> ack, will check tomorrow morning
18:51:36 <dansmith> thanks
18:52:08 <gmann> slaweq: pinged about port test in neutron channel
18:52:33 <dansmith> rosmaita: I think the one big right now for cinder stuff is that we create a server with a volume, ssh in and mkfs it, and it times out after 30 minutes without completion or some such
18:52:34 <gmann> we are close to release and if we can improve the timeout issue it will be very helpful
18:52:54 <dansmith> gmann: yeah, the timeout stuff will really help overall I think, but we have to be able to actually merge those
18:53:09 <dansmith> the server actions split has made a big difference for in the nova set I've been rechecking, fwiw
18:53:16 <dansmith> haven't seen any timeouts in the last 36 hours or so
18:53:21 <gmann> true
18:53:39 <gmann> and I am hopeful this test run concurrency increase will make job timeout improve https://review.opendev.org/c/openstack/tempest/+/887220/9
18:53:47 * dansmith nods
18:53:59 <gmann> tempest-slow job is now half of the time now by running them serially and concurrency  increase
18:54:14 <slaweq> gmann ack, I added this also to my todo list for tomorrow
18:54:17 <gmann> *running them paralley
18:54:22 <gmann> slaweq: thanks
18:54:23 <dansmith> yeah, and that's where a lot of the fails are in that series I think
18:55:30 <knikolla> anything else on the topic?
18:55:35 <dansmith> anyway, in two weeks of rechecking a series of easy nova patches, I've gotten one to merge -- that's how bad it is
18:55:57 <noonedeadpunk> oh, sounds really bad...
18:56:07 <dansmith> knikolla: that's all my actual things
18:56:10 <JayF> In a bit of a brighter note; Ironic is through the valley of our CI for now... I'm sad to hear that's not the case across everyone
18:56:11 <noonedeadpunk> We had jsut couple of timeouts, which are gone with next recheck
18:56:27 <gmann> yeah and challenge is to merge the fixes where other issue stop them to merger
18:56:29 <slaweq> dansmith yeap, I saw it also in the rechecks stats https://etherpad.opendev.org/p/recheck-weekly-summary - we need a lot of rechecks to get patches merged
18:56:35 <JayF> we have merged multiple fixes, including fixing more locking/retrying issues when testing under sqlite
18:56:38 <dansmith> slaweq: yeah :(
18:57:15 <knikolla> we're almost at time
18:57:28 <knikolla> thanks all, and thanks for all the hard work keeping the gate running
18:58:03 <spotz[m]> Thanks everyone!
18:58:22 <knikolla> #endmeeting