14:00:15 <hberaud> #startmeeting releaseteam
14:00:16 <openstack> Meeting started Fri May 21 14:00:15 2021 UTC and is due to finish in 60 minutes.  The chair is hberaud. Information about MeetBot at http://wiki.debian.org/MeetBot.
14:00:17 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
14:00:19 <hberaud> Ping list: elod armstrong
14:00:20 <openstack> The meeting name has been set to 'releaseteam'
14:00:24 <ttx> re o/
14:00:25 <hberaud> #link https://etherpad.opendev.org/p/xena-relmgt-tracking Agenda
14:00:29 <hberaud> We're way down on line 115 now
14:00:33 <hberaud> ttx: re
14:00:59 <hberaud> Will just wait a couple minutes for folks.
14:01:05 <elod> o/
14:01:54 <fungi> not sure if you were waiting for me, but if you were i'm around
14:02:06 <hberaud> thanks fungi
14:02:50 <armstrong> o/
14:03:08 <hberaud> ok let's go
14:03:11 <hberaud> #topic Review task completion
14:03:19 <hberaud> Review cycle-trailing projects to check which haven’t released yet. => Done
14:03:23 <hberaud> Here is the ML thread => http://lists.openstack.org/pipermail/openstack-discuss/2021-May/022518.html
14:03:53 <hberaud> And that was all for this week
14:04:06 <hberaud> #topic Assign R-19 tasks
14:04:24 <hberaud> Ensure that all trailing projects have been branched for the previous series.
14:04:42 <hberaud> I take it
14:04:56 <hberaud> thanks ttx
14:04:58 <ttx> I'll do the aclcheck
14:05:04 <hberaud> as usual :)
14:05:36 <hberaud> Ok next topic
14:05:41 <hberaud> #topic Review countdown email contents
14:05:48 <hberaud> https://etherpad.opendev.org/p/relmgmt-weekly-emails
14:07:24 <ttx> checking
14:08:03 <ttx> LGTM, ship it
14:08:09 <hberaud> ok thanks
14:08:19 <hberaud> I'll send it after your meeting
14:08:33 <hberaud> #topic ocata-eol status
14:08:48 <hberaud> elod: the floor is yours
14:09:10 <elod> first of all, I'm planned to continue with the delete already *-eol tagged branches
14:09:25 <hberaud> ok
14:09:34 <elod> so far ocata-eol and pike-eol branches were deleted
14:09:40 <ttx> ++
14:09:55 <elod> today will come queens, rocky and stein
14:10:09 <hberaud> cool
14:10:11 <elod> + the ones that had patches on top,
14:10:22 <elod> but agreed to delete anyway
14:10:34 <hberaud> do you need help somewhere?
14:10:53 <elod> hberaud: no, i think i can manage that with the script :)
14:11:01 <elod> but thanks :)
14:11:21 <elod> and the next is horizon's ocata-eol
14:11:23 <hberaud> I followed the ML thread and everything seems smooth
14:11:40 <elod> yes, fortunately
14:12:18 <elod> ( the horizon patch: https://review.opendev.org/c/openstack/releases/+/791702 )
14:12:25 <hberaud> yes I planed to discussed a bit about the horizon topic
14:12:27 <hberaud> https://review.opendev.org/c/openstack/releases/+/791702
14:12:40 <hberaud> *planned
14:13:04 <hberaud> Do we want to wait for Akihiro?
14:13:30 <hberaud> I think yes
14:13:57 <elod> maybe i can ping him, but Ivan, Radomir have +1'd
14:14:10 <elod> I'll ping him
14:14:21 <hberaud> As you want, but yes can't hurt to ping him
14:14:37 <hberaud> thank you
14:14:41 <elod> np
14:14:46 <hberaud> Anything else for ocata?
14:15:07 <elod> maybe,
14:15:18 <fungi> is there a feeling for when we'd want to consider integrating the manual script into release jobs?
14:15:37 <fungi> like, maybe in roughly a cycle? two?
14:15:50 <fungi> the mechanism seems to be working out well, at least
14:16:12 <elod> fungi: good question. the script requires now to enter some password, so it needs some refactor :)
14:16:31 <fungi> sure, presumably we'd authenticate it the same way we do branch creation
14:16:50 <hberaud> AFAIK we didn't considered to integrate this script into a job but why not
14:17:10 <fungi> i just hate to think of release managers constantly manually running the script in coming years
14:17:23 <elod> true
14:17:33 <elod> I'll add that to my todo list :)
14:17:35 <fungi> it's not urgent, just something to keep in mind as you're grooming the rest of the release automation over time
14:18:12 <hberaud> it can be a hook triggered by eol tag or something
14:18:27 <hberaud> in our machinery
14:18:30 <elod> that should do the trick
14:18:38 <fungi> also are externally added periodic jobs being cleaned up once the eol branches get deleted?
14:18:57 <fungi> i do still see over a hundred job failure notifications every day to the stable list
14:19:24 <elod> i've discovered some failing periodic job,
14:19:26 <fungi> if there are project-config changes which need reviewing to remove some, please give me a heads up and i'm happy to take a look
14:19:35 <elod> for projects that wanted to use neutron's stable/ocata,
14:19:46 <elod> which was deleted ~ a week ago
14:19:57 <elod> I've sent a mail to the team
14:20:41 <elod> fungi: thanks, will remember if we need such changes
14:21:00 <fungi> cool
14:21:04 <hberaud> where are defined periodic jobs? in project-config too?
14:21:14 <fungi> many are
14:21:34 <fungi> if the jobs are defined in the repositories themselves then they disappear when the branches do, of course
14:21:41 <hberaud> yes
14:21:43 <fungi> defined or added to pipelines
14:22:25 <armstrong> ttx I will like to assit in your task for this week, of you don’t mind
14:22:27 <elod> (i've added a reminder to look into projects-config's periodic job definitions :))
14:22:54 <ttx> armstrong: ok let em see when i could do it
14:23:25 <ttx> armstrong: would Wednesday 1300utc or 13:30utc work for you?
14:23:48 <armstrong> Ok sounds good
14:23:59 <ttx> which one? 13?
14:24:11 <ttx> I'm trying to block the time to be sure
14:24:13 <armstrong> 13
14:24:21 <ttx> \ok noted, I'll ping you here
14:24:29 <armstrong> Ok
14:25:05 <elod> and about general/"mass" ocata-eol - I will get there next week I think to check the activities in projects, and will propose ocata-eol patches (i guess multiple ones?) + mail to ML
14:25:17 <elod> if this is OK for you ^^^^
14:25:38 <hberaud> WFM
14:26:10 <elod> i think that's it for ocata-eol/*-eol
14:26:17 <hberaud> do you plan to propose patch per team?
14:26:30 <hberaud> (multiple ones)
14:26:40 <elod> hberaud: i think that would be the best
14:26:45 <hberaud> WFM
14:27:05 <elod> ++
14:28:23 <hberaud> fungi: I just have a question concerning the periodic jobs, can we identify if they are bound to series?
14:28:51 <fungi> i'd have to look at some example failures
14:29:05 <fungi> probably easiest to approach from concrete examples
14:29:11 <elod> there is a periodic template, with the branches listed in it
14:29:22 <elod> (actually multiple templates, but anyway)
14:29:26 <hberaud> I suppose that's depends on the job, some could be for all series, some could be specific
14:29:28 <hberaud> ok
14:29:55 <hberaud> thanks
14:30:13 <elod> until ocata is not fully EOL we should not touch that, only if there are such that would run especially only against branches that are already eol'd
14:30:23 <fungi> yeah, part of the challenge with project-templates is that if you remove a branch from a multi-branch template then you stop running it for all projects rather than just those which have eol'd those branches, and if you remove the template from the project then you likely stop running the jobs for active branches too
14:30:44 <hberaud> I see
14:30:50 <hberaud> elod: yes
14:30:51 <fungi> an alternative would be to make branch-scoped project-templates and add or remove them separately
14:31:10 <fungi> similar to how we do with the pti jobs
14:31:30 <fungi> (victoria template, wallaby template, and so on)
14:31:35 <hberaud> I see
14:32:05 <hberaud> branch-scoped could be a good thing
14:32:13 <elod> i think ocata can be removed from the branches list later on, that should not disturb other branches. but I might miss something
14:32:28 <elod> (from the periodic templates)
14:32:28 <fungi> so depending on where/how the jobs are currently getting added, it might not be a simple matter of just deleting some lines, we may need to refactor how that's being done instead
14:32:43 <fungi> which is why i say specific examples matter
14:32:59 <hberaud> ok
14:33:47 <hberaud> thanks for these details
14:33:58 <fungi> also the job failures are obviously not just noise on a mailing list, they represent a lot of wasted ci resources
14:34:13 <fungi> which is a big part of why i keep checking up on the situation
14:34:23 <elod> that's true
14:34:25 <hberaud> indeed
14:34:53 <elod> actually right now most of the failures are the non-SNI client related ones
14:34:59 <fungi> if we expect the jobs to start working again at some point then it's probably okay to keep running them for now, but if they're never going to work again we should stop running them
14:35:30 <elod> hmmm. yes.
14:35:53 <fungi> is someone working on getting a workaround figured out for the sni support issue on those?
14:35:55 <elod> there are projects that are quite abandoned, so maybe we can remove the periodic for those
14:36:23 <elod> (like *-powervm)
14:36:48 <hberaud> that could be a way to start to reduce the resources usage
14:36:55 <elod> fungi: It's on my todo as well o:)
14:37:14 <elod> fungi: and did some fix already, but for grenade jobs
14:37:55 <fungi> thanks!
14:38:24 <elod> anyway, I'll propose some periodic removal patches as well for inactive projects (hope someone will review + approve them)
14:38:45 <fungi> i'm happy to review any for project-config and openstack-zuul-jobs
14:38:59 <fungi> just let me know when you push them so i can prioritize
14:38:59 <hberaud> +1
14:39:12 <elod> fungi: nice, thanks!
14:39:31 <elod> fungi: some are i guess in the project's repository
14:39:36 <elod> but we will see
14:40:07 <fungi> yeah, if you can put together a list of the ones which are inside abandoned projects, i can also come up with a strategy there
14:40:44 <elod> fungi: ok, thanks!
14:40:45 <fungi> the opendev sysadmins are free to exercise control over the repository hosting to remove job configuration when it's problematic
14:41:36 <elod> sure, then it won't be a problem :)
14:42:04 <elod> maybe i can use my stable-maint-core power as well, but we will see
14:43:19 <hberaud> I think that we can continue on the next topic
14:43:25 <hberaud> #topic train-em status
14:43:36 <elod> https://review.opendev.org/q/topic:%2522train-em%2522+status:open
14:43:46 <elod> it's less than a page \o/
14:43:47 <elod> :]
14:43:54 <hberaud> :)
14:44:17 <elod> some have -1 that we should wait for the teams
14:44:23 <hberaud> I don't expect PTL responses for a couple of them
14:44:51 <elod> yes, some don't seem to get responses
14:45:04 <fungi> are those situations we need to relay to the tc?
14:45:20 <hberaud> let's wait one more week for those without response
14:45:58 <fungi> projects not at least acknowledging release changes and blocking series transitions due to inactivity are probably signs the project is mostly defunct
14:46:03 <hberaud> hm... usually, for the current series we force patches without response (depends-on the topic of these patches)
14:46:04 <elod> i don't know whether we should relay to the tc or simply just force the train-em transition there
14:46:16 <elod> oh, i see
14:46:30 <hberaud> I think that in this case we should force
14:47:04 <elod> what fungi says is right, though
14:47:19 <hberaud> these project should follow the life cycle of the series
14:47:23 <hberaud> yes
14:47:25 <elod> the question is whether the projects just missed the release patch,
14:47:26 <fungi> yeah, not saying to ask the tc for permission, just letting them know so they can check on whether the project is still active
14:47:37 <elod> or inactive...
14:47:50 <ttx> need to run, ping me if you need me
14:48:05 <elod> i think it worth to ping the teams on ML / IRC first
14:48:06 <fungi> if projects are really dead the tc can take over and retire them, which means less work for the release team
14:48:18 <fungi> (in the long run anyway)
14:48:32 <hberaud> but yes I think we more need to inform the TC to decide the status of these project for the current or the next series
14:48:56 <hberaud> ttx: ack, thanks
14:49:01 <elod> (projects like, keystone, swift, monasca, ironic, etc)
14:49:31 <hberaud> the plan could be 1) ping the team 2) inform the TC 3) force the patches
14:49:50 <elod> let's start with the 1st option :)
14:49:58 <fungi> also remember keystone doesn't have a ptl, they're supposed to have an active release liaison though under the dpl model
14:50:10 <hberaud> right
14:50:24 <hberaud> as oslo
14:50:27 <fungi> yes
14:50:29 <elod> I'll ping the teams
14:50:44 <hberaud> thanks elod
14:51:00 <elod> no problem :)
14:51:18 <hberaud> Anything else for train-em?
14:51:35 <elod> and we will see whether we need to inform TC regarding any of the projects
14:51:55 <elod> hberaud: nothing from my side
14:52:01 <hberaud> I already discussed with the TC about some of them at the end of wallaby
14:52:21 <elod> oh, good
14:52:32 <hberaud> at the end of each cycle we check the project/release activity with the TC
14:52:54 <fungi> but yeah, continuing to communicate missed deadlines will help them know if the situations change
14:53:00 <hberaud> they are already aware for some of them
14:53:08 <hberaud> yes
14:54:07 <hberaud> ok thanks
14:54:15 <hberaud> #topic Open Floor
14:54:27 <hberaud> Anything else to discuss today?
14:54:47 <elod> nothing else from me :X
14:55:00 <hberaud> +1
14:57:00 <hberaud> OK, thanks everyone. Let's wrap up.
14:57:02 <hberaud> #endmeeting