14:00:56 <elodilles> #startmeeting releaseteam
14:00:56 <opendevmeet> Meeting started Fri Sep 23 14:00:56 2022 UTC and is due to finish in 60 minutes.  The chair is elodilles. Information about MeetBot at http://wiki.debian.org/MeetBot.
14:00:56 <opendevmeet> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
14:00:56 <opendevmeet> The meeting name has been set to 'releaseteam'
14:00:59 <hberaud> o/
14:01:15 <armstrong> o/
14:01:22 <elodilles> hello all o/
14:01:27 <ttx> o/
14:01:33 <elodilles> #link https://etherpad.opendev.org/p/zed-relmgt-tracking
14:01:58 <elodilles> we are at L408
14:02:15 <elodilles> almost at the bottom :-o
14:02:31 <elodilles> let's start with the topics
14:02:42 <elodilles> #topic Review task completion
14:03:05 <elodilles> 1st task was:
14:03:09 <elodilles> #info Process any standing RC1 deadline exceptions.
14:03:35 <elodilles> https://review.opendev.org/q/topic:zed-rc1-deadline+is:open
14:04:11 <elodilles> these were mostly the ones that had broken gates
14:04:47 <elodilles> during the week we agreed that we should continue with a 'formal' release from this
14:05:11 <elodilles> * these
14:05:27 <elodilles> I'll merge them after the meeting if no objection
14:06:05 <ttx> The barbican one whould wait
14:06:09 <elodilles> except barbican, as it has still -1 on it
14:06:16 <elodilles> yepp
14:06:40 <ttx> that would be the last standing one, unless dmendiza[m] has an update
14:06:52 <elodilles> yepp
14:07:10 <ttx> We are super late for this one so if we don;t hear on Monday I'd consider bypassing it
14:07:31 <opendevreview> Merged openstack/releases master: Creating stable zed for swift  https://review.opendev.org/c/openstack/releases/+/858301
14:08:00 <elodilles> ttx: ack, i'll reach out to the team and inform them about it
14:08:28 <elodilles> so that's about this task
14:08:38 <elodilles> 2nd task was:
14:08:51 <elodilles> #info Gracefully release tempest plugins with latest changes available. (elod)
14:08:58 <elodilles> https://review.opendev.org/q/topic:zed-tp-latest
14:09:31 <elodilles> we have two -1s
14:09:43 <elodilles> ~half were accepted and merged
14:10:08 <elodilles> i'll merge the ones without response after the meeting
14:10:19 <ttx> +1
14:10:55 <elodilles> 3rd task
14:11:01 <elodilles> #info On Monday, generate stable branches for all cycle deliverables that are still missing one. (hberaud)
14:11:15 <elodilles> https://review.opendev.org/q/topic:zed-missing-stable-branches (some deliverables are listed in deliverables that are still missing stable branches, however, those have also RC1 pending patches, so I ignored them)
14:11:32 <hberaud> I think we can now ignore my comment
14:11:38 <elodilles> hberaud: ack
14:11:57 <opendevreview> Merged openstack/releases master: Creating stable zed for python-openstackclient  https://review.opendev.org/c/openstack/releases/+/858291
14:11:58 <elodilles> but then I'll propose the stable-branch cutting patches as well
14:12:09 <elodilles> i guess we'll need some
14:12:48 <hberaud> ack
14:12:51 <elodilles> (the rest of the sub-points were only instructions for this task)
14:13:09 <elodilles> info After all the projects enabled in devstack by default have been branched, we can engage with the QA, I18n and Requirements PTLs to finalize the stable branch setup. (elod)
14:13:48 <elodilles> i haven't processed this one as we still have some unbranched projects
14:14:05 <elodilles> which will be branched after the above task is finished
14:14:18 <elodilles> so i copy this task to next week as well
14:15:01 <elodilles> otherwise, should be OK
14:15:14 <elodilles> next task:
14:15:22 <elodilles> #info Ensure that all projects that are publishing release notes have the notes link included in their deliverable file. (hberaud)
14:15:34 <elodilles> the patch was proposed:
14:15:36 <elodilles> https://review.opendev.org/c/openstack/releases/+/858866
14:16:26 <elodilles> this is usually not complete for some time after RC1
14:16:43 <elodilles> as this needs all the reno-$series patches to get merged
14:17:11 <elodilles> i see some -1s on the patch already from PTLs
14:17:29 <elodilles> should we wait for an update for the patch?
14:18:05 <elodilles> or merge this today and propose a follow up later on?
14:18:19 <ttx> checking what the -1s are for
14:18:44 <elodilles> missing reno links basically
14:19:07 <ttx> ah so we should regenerate the patch right
14:19:25 <elodilles> yes, maybe that's the easiest
14:19:40 <hberaud> we could add the missing links through a follow up
14:19:48 <elodilles> and i'm sure we can re-iterate this task some time later (as i did in last cycle)
14:19:50 <hberaud> as we did with prvious series
14:20:50 <elodilles> what about regenerating the links as a 2nd PS for this one, and we merge it ASAP?
14:21:00 <hberaud> as you want
14:21:09 <hberaud> both wfm
14:21:13 <elodilles> and later on creating a follow-up
14:21:22 <elodilles> if needed
14:21:31 <hberaud> ok I'll update the patch
14:21:40 <elodilles> (just to not merge with lots of -1 on it ;))
14:21:53 <elodilles> hberaud: sorry, and thanks! o:)
14:21:57 <hberaud> np
14:22:47 <elodilles> the last task is again, not a task, but a note: Let cycle-with-rc projects iterate on RCs as needed. The final release candidate for each project needs to be prepared at least one week before the final release date.
14:24:23 <elodilles> so this means the final RC iterations (if needed) needs to be proposed next week
14:24:59 <elodilles> ok, let's move to the next topic:
14:25:07 <elodilles> #topic Assign R-1 tasks
14:25:23 <elodilles> please add your name if you want to work with any of the tasks
14:27:23 <elodilles> Ok, as I see, all taken
14:27:26 <elodilles> thanks :)
14:27:46 * ttx looks up what he signed up for
14:27:59 <elodilles> :]
14:28:16 <elodilles> #topic Review countdown email contents
14:28:21 <ttx> looks like the deadline is Friday, so "day before deadline" is Thursday
14:28:33 <elodilles> https://etherpad.opendev.org/p/relmgmt-weekly-emails
14:28:36 <elodilles> please review ^^^
14:28:43 <elodilles> ttx: yepp
14:29:35 <ttx> ah no the deadline is Sep 29 on the email
14:29:44 <ttx> so I'll propose last-minute ones on the 28th
14:29:53 <elodilles> yes, just wanted to correct,
14:30:21 <elodilles> that we discussed this some time ago that it should be 29th September O.o
14:30:38 <ttx> ack
14:31:00 <ttx> email lgtm
14:31:20 <opendevreview> Hervé Beraud proposed openstack/releases master: Add release note links for zed  https://review.opendev.org/c/openstack/releases/+/858866
14:31:39 <hberaud> release notes links updated ^
14:31:54 <elodilles> ttx hberaud : ack, thanks!
14:32:07 <elodilles> will send + will review :)
14:32:13 <hberaud> mail LGTM
14:32:17 <elodilles> ++
14:32:30 <elodilles> #topic Open Discussion
14:32:43 <elodilles> anything to mention here?
14:33:06 <ttx> do we think we captured all the oslo.db issues?
14:33:14 <ttx> I saw a few late mentions pass by while I was traveling
14:34:03 <elodilles> good question
14:34:19 <elodilles> at least we know about some
14:35:23 <elodilles> watcher and ec2-api were mentioned in the last mails: https://lists.openstack.org/pipermail/openstack-discuss/2022-September/030493.html
14:36:19 <ttx> we might want to chase a fix -- although I'll admit I'll have trouble to have cycles to do that next week
14:36:42 <elodilles> for watcher i see a patch: https://review.opendev.org/c/openstack/watcher/+/858032
14:36:54 <elodilles> but it seems zuul ignores that patch :-o
14:37:55 <ttx> hmm
14:37:57 <elodilles> ah, broken gate: https://review.opendev.org/c/openstack/watcher/+/857393
14:38:51 <elodilles> strange that on master it seems fine
14:39:06 <elodilles> and its already branched
14:39:39 <ttx> and the failures look functional
14:40:06 <elodilles> yes
14:40:33 <ttx> I guess it could all be linked to DB access failures
14:40:52 <elodilles> ec2-api, the same, here is the fix: https://review.opendev.org/c/openstack/ec2-api/+/857880
14:40:56 <elodilles> but the gate is broken
14:41:48 <ttx> more data for our "abandoned" report
14:42:34 <ttx> At least ec2api fails in a clearer manner
14:42:36 <elodilles> exactly..
14:42:42 <ttx> TypeError: missing a required argument: 'public_key'
14:43:06 <ttx> File "/home/zuul/src/opendev.org/openstack/ec2-api/ec2api/api/key_pair.py", line 88, in create_key_pair
14:43:26 <ttx> I'll have a quick look in meeting in case I can fix it quick
14:43:28 <ttx> 17 min
14:44:23 <iurygregory> hberaud, https://review.opendev.org/c/openstack/releases/+/858866 is missing ironic-inspector (or at least I couldn't find)
14:44:51 <elodilles> ttx: there is a patch that tried already to fix the key issue i think: https://review.opendev.org/c/openstack/ec2-api/+/858143
14:45:50 <ttx> stephenfin to the rescue again
14:47:08 <elodilles> anyway, i've added it on our tasks list for R-1
14:47:35 <elodilles> i'll try to follow up on them, but any help is appreciated
14:47:50 <stephenfin> That needs tests. The issue is that nova no longer generates keypairs for you so we need to do it ourselves, client-side. I've yet to verify that the SSH keys I'm generating there match the format of what nova was generating (since nova was using paramiko but we're using the lower-level cryptography lib)
14:48:02 <elodilles> i really hope these are all that is related to oslo.db 12.1.0 for now :)
14:48:06 <stephenfin> Also, some missing imports, clearly (I was testing in the terminal and copy/pasting)
14:48:25 <ttx> yeah this is non-trivial really
14:48:27 <hberaud> iurygregory: indeed, I just regenerated it with our script, so, something else is at work, which prevent the generation of this link.
14:49:00 <stephenfin> elodilles: Sorry oslo.db 12.1.0 has been such a pain. It'll be less painful that SQLAlchemy 2.0 landing on with no warning though. At least not we know where all the tech debt is :)
14:49:07 <stephenfin> s/not/now/
14:49:30 <hberaud> iurygregory: https://docs.openstack.org/releasenotes/ironic-inspector/zed.html => 404
14:49:36 <ttx> if all the codebases were functional it would not have been that much of an issue
14:49:40 <iurygregory> =O
14:49:49 <iurygregory> let me check things in inspector
14:49:55 <ttx> its just that it woke up areas of the code that were happily sleeping
14:49:59 <elodilles> stephenfin: fingers crossed :) and thanks for all the work you done to fix all the found issues!!!
14:50:00 <hberaud> I think its why the links is not added
14:50:23 <hberaud> /link/
14:51:01 <fungi> iurygregory: hberaud: someone brought one up yesterday where the release notes promote job hit a failure during rsync, probably intersected with a change for another branch trying to lock the same tree for writing
14:51:22 <hberaud> I see
14:51:23 <fungi> in that case we simply reenqueued the change into the promote pipeline
14:51:24 <iurygregory> fungi, ohhh
14:51:38 <elodilles> and today we were informed with the same issue
14:52:47 <fungi> if someone can put together a list of which builds failed, i'm happy to do the reenqueuing (but be mindful that doing so may roll back documentation if a later change published newer documentation and didn't update the release notes)
14:52:56 <elodilles> and i saw that the reno-zed patch was merged later than the reno updating job ran
14:52:58 <iurygregory> https://docs.openstack.org/releasenotes/ironic-inspector/
14:53:03 <iurygregory> yeah I don't see Zed .-.
14:53:17 <elodilles> so after triggering the promote job with a patch, it got updated
14:53:45 <iurygregory> https://opendev.org/openstack/ironic-inspector/commit/64c881b20fdf694a719e0c7793e72f0345679623 so this one failed to promote I think
14:55:35 <fungi> yeah, the options are to either reenqueue the change into promote if there wasn't any newer change which did docs builds for that branch, or merge another change to the branch which triggers a release notes build
14:56:05 <iurygregory> I don't think we have any changes we can merge in inspector atm, let me check
14:56:57 <iurygregory> ok, there is one, let's see :D
14:57:03 <fungi> this was the promote buildset for the commit you linked: https://zuul.opendev.org/t/openstack/buildset/3cf3c8863f0e45ba9370609ac2c4c2b5
14:57:17 <fungi> and it ran promote-openstack-releasenotes
14:57:20 <fungi> successfully
14:57:28 <iurygregory> O.o
14:58:10 <fungi> https://zuul.opendev.org/t/openstack/build/edd6954962d646a39836faeddb3ade6f/console#1/0/28/localhost
14:58:22 <fungi> <<CHANGED>>>f+++++++++ zed.html
14:58:44 <iurygregory> enr .-. (no idea why it doesn't show...)
14:59:36 <fungi> there are a couple of possibilities i can look into. either another build later un-published that file, or our afs vos release cron is failing to publish that update to the read-only replicas
14:59:41 <ttx> posted a new patch fixing the imports for ec2api -- no idea if it actually works though
14:59:50 <ttx> https://review.opendev.org/c/openstack/ec2-api/+/858143
15:00:19 <iurygregory> I'm merging https://review.opendev.org/c/openstack/ironic-inspector/+/859016
15:00:32 <iurygregory> let's see if with the new build it will show-up
15:01:12 <elodilles> meanwhile, we've run out of meeting time, so let's close the meeting, but please continue the debugging if needed o:)
15:01:24 <elodilles> thanks everyone for joining o/
15:01:28 <fungi> /afs/.openstack.org/docs/releasenotes/ironic-inspector/ (the writeable volume) does not include a zed.html, so i think it got unpublished
15:01:29 <elodilles> #endmeeting