14:00:56 #startmeeting releaseteam 14:00:56 Meeting started Fri Sep 15 14:00:56 2023 UTC and is due to finish in 60 minutes. The chair is elodilles. Information about MeetBot at http://wiki.debian.org/MeetBot. 14:00:56 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 14:00:56 The meeting name has been set to 'releaseteam' 14:01:00 o/ 14:01:16 #link https://etherpad.opendev.org/p/bobcat-relmgt-tracking 14:01:20 o/ 14:01:20 Merged openstack/releases master: Release zaqar RC1 for 2023.2 Bobcat https://review.opendev.org/c/openstack/releases/+/894728 14:01:22 Merged openstack/releases master: Release zaqar-ui RC1 for 2023.2 Bobcat https://review.opendev.org/c/openstack/releases/+/894727 14:01:25 Merged openstack/releases master: Release manila RC1 for 2023.2 Bobcat https://review.opendev.org/c/openstack/releases/+/894668 14:01:27 Merged openstack/releases master: Release barbican RC1 for 2023.2 Bobcat https://review.opendev.org/c/openstack/releases/+/894644 14:01:29 Merged openstack/releases master: Release kuryr-kubernetes RC1 for 2023.2 Bobcat https://review.opendev.org/c/openstack/releases/+/894663 14:01:37 Merged openstack/releases master: Release designate RC1 for 2023.2 Bobcat https://review.opendev.org/c/openstack/releases/+/894654 14:01:39 Merged openstack/releases master: Release kuryr-libnetwork RC1 for 2023.2 Bobcat https://review.opendev.org/c/openstack/releases/+/894664 14:01:41 Merged openstack/releases master: Release octavia RC1 for 2023.2 Bobcat https://review.opendev.org/c/openstack/releases/+/894695 14:01:44 Merged openstack/releases master: Release octavia-dashboard RC1 for 2023.2 Bobcat https://review.opendev.org/c/openstack/releases/+/894694 14:01:46 Merged openstack/releases master: Release senlin RC1 for 2023.2 Bobcat https://review.opendev.org/c/openstack/releases/+/894710 14:01:48 Merged openstack/releases master: Release ceilometer RC1 for 2023.2 Bobcat https://review.opendev.org/c/openstack/releases/+/894648 14:01:50 Merged openstack/releases master: Release aodh RC1 for 2023.2 Bobcat https://review.opendev.org/c/openstack/releases/+/894643 14:01:53 Merged openstack/releases master: Release blazar-nova RC1 for 2023.2 Bobcat https://review.opendev.org/c/openstack/releases/+/894646 14:01:55 Merged openstack/releases master: Release adjutant RC1 for 2023.2 Bobcat https://review.opendev.org/c/openstack/releases/+/894639 14:01:58 we are at R-3, line 411, very close to the bottom of the etherpad :-o 14:02:03 Merged openstack/releases master: Release monasca-agent RC1 for 2023.2 Bobcat https://review.opendev.org/c/openstack/releases/+/894674 14:02:05 Merged openstack/releases master: Release monasca-notification RC1 for 2023.2 Bobcat https://review.opendev.org/c/openstack/releases/+/894677 14:02:07 Merged openstack/releases master: Release monasca-api RC1 for 2023.2 Bobcat https://review.opendev.org/c/openstack/releases/+/894675 14:02:56 and let's start! 14:03:07 #topic Review task completion 14:03:13 Merged openstack/releases master: Release monasca-thresh RC1 for 2023.2 Bobcat https://review.opendev.org/c/openstack/releases/+/894679 14:03:20 "Process any remaining library branching exception (all)" 14:03:34 https://review.opendev.org/q/topic:bobcat-stable-branches+is:open 14:03:37 Merged openstack/releases master: Release monasca-ui RC1 for 2023.2 Bobcat https://review.opendev.org/c/openstack/releases/+/894680 14:03:39 Merged openstack/releases master: Release monasca-persister RC1 for 2023.2 Bobcat https://review.opendev.org/c/openstack/releases/+/894678 14:03:41 Merged openstack/releases master: Release monasca-events-api RC1 for 2023.2 Bobcat https://review.opendev.org/c/openstack/releases/+/894676 14:04:13 hmmm, we have still some branching patches open 14:06:05 I think we should merge them, except the python-barbicanclient one, but we have to ping the team there as it has to be merged ASAP 14:06:35 wfm 14:07:52 OK, I've added my +2s there 14:08:02 hberaud: can you quickly check them? 14:08:07 Yasufumi Ogawa proposed openstack/releases master: Release tacker RC1 for 2023.2 Bobcat https://review.opendev.org/c/openstack/releases/+/894717 14:08:07 yeah 14:09:34 done 14:09:39 thanks 14:09:50 np 14:09:55 next task then! 14:10:05 "on monday force the milestone 3 patches that are still opened (all)" 14:10:11 https://review.opendev.org/q/topic:bobcat-milestone-3+is:open 14:10:35 none is open, so we are done with this task 14:11:01 "On the Monday, generate release requests for all deliverables that have do not have a suitable candidate yet (elod)" 14:11:23 this had multiple sub tasks ^^^ 14:11:28 "Branch devstack-plugin-* delivrables" 14:11:34 https://review.opendev.org/c/openstack/releases/+/894743 14:11:51 this is merged, so we are good 14:12:02 "cycle-with-intermediary deliverables" 14:12:07 https://review.opendev.org/c/openstack/releases/+/893418 14:12:25 this is the only one still open ^^^ 14:12:33 python-openstackclient 14:13:10 I think we are late so I'd suggest to force it 14:13:18 i had a comment on the patch as it is there are some merged patches that are not part of the release 14:13:37 s/it is// 14:14:01 Merged openstack/releases master: Release sahara RC1 for 2023.2 Bobcat https://review.opendev.org/c/openstack/releases/+/894708 14:14:03 Merged openstack/releases master: Release sahara-plugin-spark RC1 for 2023.2 Bobcat https://review.opendev.org/c/openstack/releases/+/894705 14:14:06 Merged openstack/releases master: Release sahara-plugin-cdh RC1 for 2023.2 Bobcat https://review.opendev.org/c/openstack/releases/+/894703 14:14:08 Merged openstack/releases master: Release sahara-plugin-storm RC1 for 2023.2 Bobcat https://review.opendev.org/c/openstack/releases/+/894706 14:14:17 I can ping gtema about the osc release 14:14:19 i was thinking whether we should update the hash before we proceed with the release 14:14:35 frickler: thx 14:14:39 frickler: that would be awesome 14:14:42 elodilles, sorry, missed to response 14:14:59 Merged openstack/releases master: Release sahara-plugin-ambari RC1 for 2023.2 Bobcat https://review.opendev.org/c/openstack/releases/+/894702 14:15:02 Merged openstack/releases master: Release sahara-plugin-vanilla RC1 for 2023.2 Bobcat https://review.opendev.org/c/openstack/releases/+/894707 14:15:04 Merged openstack/releases master: Release sahara-plugin-mapr RC1 for 2023.2 Bobcat https://review.opendev.org/c/openstack/releases/+/894704 14:15:06 Merged openstack/releases master: Release sahara-image-elements RC1 for 2023.2 Bobcat https://review.opendev.org/c/openstack/releases/+/894701 14:15:09 Merged openstack/releases master: Release sahara-extra RC1 for 2023.2 Bobcat https://review.opendev.org/c/openstack/releases/+/894700 14:15:10 more or less every day release change is not merged there is a big chance for new changes to come 14:15:37 there was nothing urgent merged after change was created 14:15:37 gtema: now repos should have feature freeze on them o:) 14:16:14 but we have branched already 14:16:30 gtema: not yet 14:16:36 gtema: good point 14:16:48 gtema: this is the patch: https://review.opendev.org/c/openstack/releases/+/893418 14:16:55 python-openstackclient 14:17:46 ehm, I was sure I have seen a change for branching 14:18:24 Sylvain Bauza proposed openstack/releases master: Release placement RC1 for 2023.2 Bobcat https://review.opendev.org/c/openstack/releases/+/894698 14:18:31 you are producing so many changes in last days that I completely lost overview 14:19:03 gtema: a general comment: it is good to avoid merging patches (especially bigger changes) between FF/branching and the final release, to avoid backporting conflicts 14:19:20 https://review.opendev.org/c/openstack/releases/+/893619 - I was sure it was also for osc itself 14:20:40 gtema: 1st we release non-client libraries then client-libraries and other cycle-with-intermediary projects. but i know, many patches o:) 14:20:40 Artem Goncharov proposed openstack/releases master: Release python-openstackclient for Bobcat https://review.opendev.org/c/openstack/releases/+/893418 14:20:56 ok, updated the hash 14:21:02 thanks! 14:21:09 there was anyway nothing big landed in the meanwhile 14:21:22 wlcm and one more time sorry for missing to response 14:21:25 respond 14:21:34 no worries 14:21:42 i'll add the branching to it, too 14:22:26 Merged openstack/releases master: Release sahara-dashboard RC1 for 2023.2 Bobcat https://review.opendev.org/c/openstack/releases/+/894699 14:23:29 Elod Illes proposed openstack/releases master: Release python-openstackclient for Bobcat https://review.opendev.org/c/openstack/releases/+/893418 14:23:33 gtema: ^^^ 14:23:59 approved 14:24:08 and that's all, i try to not bother you anymore o:) 14:24:16 thanks! \o/ 14:24:55 let's wait until the gate finishes with a Verified+1 and we can merge then 14:25:00 hberaud: ^^^ 14:25:05 ack 14:25:24 cool. let's move on then 14:25:35 "cycle-with-rc that are not trailing deliverables and that have not done a RC1 yet" 14:25:43 #link https://review.opendev.org/q/topic:bobcat-rc1-deadline 14:26:09 most of them have been merged \o/ 14:26:40 some -1s, that hopefully will be sorted out in a couple of days 14:26:58 https://review.opendev.org/c/openstack/releases/+/894717 is weird, it has PTL -1 but still PTL-approved +1 14:27:30 yeah, sometimes there are anomalies around PTL-Approved :S 14:27:45 usually with another comment it is fixing itself 14:28:23 ok, nevermind, then 14:28:39 hmmm. now that i'm looking at the patch, i guess it's because PTL has pushed the patch 14:29:06 can probably look at the build log for that job to figure out why it decided that was an approval, or improve the job's logging if not 14:29:10 so it's a small bug in our job :P 14:29:34 aha, yes i bet that overrides the vote checking 14:30:00 some finetune will be needed there in the job script then :S 14:30:14 anyways 14:30:31 note: we had to merge some release patch where there are gate issues, sadly :/ 14:31:05 but that's the way we handled them in last cycles :/ 14:31:28 mostly monasca and sahara deliverables related 14:32:23 otherwise as I said, most of the rc1 releases are out! \o/ 14:32:31 \o/ 14:32:49 next task: 14:32:57 "missed client library releases (repos that did not have any merged changes during Bobcat cycle)" 14:33:21 i added this during the week, as i found some missing releases ^^^ 14:33:28 everything is now merged 14:33:41 https://review.opendev.org/q/topic:bobcat-client-lib-no-changes 14:34:30 this was originally a sub task of R-5: 'Evaluate any non-client libraries that did not have any change merged over the cycle....' 14:35:29 Merged openstack/releases master: [mistral] Create 2023.2 branch for client and non-client libs https://review.opendev.org/c/openstack/releases/+/894771 14:35:31 Merged openstack/releases master: [murano] Create 2023.2 branch for client and non-client libs https://review.opendev.org/c/openstack/releases/+/894780 14:35:34 there is a sentence on the process page, something like if we are not changing release model, then release them 14:35:47 I see 14:36:13 so we have to remind ourselves in next release to not forget this step o:) 14:36:42 :) 14:37:12 anyway, as you said, patches merged, so this is done! 14:37:29 let's move on: 14:37:32 "track the oslo.db and castellan isues to get it sorted out as soon as possible (hberaud)" 14:37:56 hberaud: can you tell us about the status here? 14:38:18 so, I think we won't see these releases moves on before the final release.... 14:38:56 neither of them? 14:39:10 a mailing thread is currently happening and are related more or less to these 2 deliverables issues 14:39:12 yeah 14:39:21 :/ 14:39:35 both are related to services who are not compat with the new versions 14:39:44 so services needs changes 14:40:00 and i guess we are late for that :/ 14:40:02 (mostly manila and masakari) 14:40:03 I've learned since last week that oslo.db >= 13 requires sqla >= 2.0, but does not specify that in its requirements 14:40:17 yeah 14:40:22 :S 14:40:46 this mail summarize well the situation https://lists.openstack.org/pipermail/openstack-discuss/2023-September/035100.html 14:40:52 so I've also alerted the TC of this 14:40:59 thanks 14:41:28 heat is also very far from adopting their db code it seems 14:41:49 thanks 14:41:55 so I guess we need to handle these similarly like we did in Antelope and Zed: we reverted the merged patches and did another releases, to avoid confusions 14:42:17 yeah, heat, aodh, masakari, manila 14:42:23 in this case oslo.db and castellan need the reverts 14:42:56 hm... for oslo.db I think that would require several reverts... 14:43:04 not sure this is a good approach 14:43:05 castellan is just a case of bad timing, they released early in the cycle, but then it got forgotten 14:43:23 for oslo.db I think they should be doing things in a feature branch really 14:44:05 frickler: that's a good thing to think about in the future 14:44:07 and requirements could be pined on the service side 14:45:01 yes, we have quite many oslo releases: 13.0.0, 13.0.1, 13.1.0, 14.0.0... and all of them not used during the cycle but listed among Bobcat's releases :S ouch :S 14:45:46 yeah 14:45:49 so what should we do then? :/ 14:45:52 Merged openstack/releases master: [sahara] Create 2023.2 branch for client and non-client libs https://review.opendev.org/c/openstack/releases/+/894777 14:45:54 Merged openstack/releases master: [monasca] Create 2023.2 branch for client and non-client libs https://review.opendev.org/c/openstack/releases/+/894781 14:45:57 Merged openstack/releases master: [glance] Create 2023.2 branch for client and non-client libs https://review.opendev.org/c/openstack/releases/+/894779 14:45:59 hberaud: pinning on the service side doesn't work for a distro that can only have one release of a lib, ask zigo ;) 14:46:12 frickler: I see 14:46:40 elodilles: I'd suggest to update the Bobcat list 14:46:50 (manually?) 14:47:03 i'm not fan of that :/ 14:47:10 to reflect the reality 14:47:11 we don't want to change history 14:47:14 me too 14:47:20 and then have no oslo.db in bobcat? or the latest from 2023.1? 14:47:31 yeah 14:47:39 I agree with frickler: everything (and I really mean *EVERYITHING*) must be co-installable. 14:47:40 (the last of antelope) 14:48:24 is there any other option? :/ 14:48:52 well a large revert spam and redo all those things on a feature branch? 14:48:57 Yeah, reverse changes in these libs, and re-release. 14:48:59 elodilles: merge stephenfin's patches on the service side 14:49:10 That's another option (a bad one, I admit). 14:49:16 hberaud: there are none for heat 14:49:45 ah 14:49:53 And yeah, also that: fix problematic services. 14:50:42 stephenfin: what do you think about reverting the oslo.db changes to rewind before the version 13.0.0? 14:50:43 we can do that next cycle, merge oslo.db bump early and then mark failing projects as release questionable instead 14:52:25 to revert that amount of patches is painful, but maybe this is the best approach + what frickler suggests ^^^ 14:53:24 you suggest to revert during C? 14:53:34 nope 14:53:40 revert ASAP, 14:53:43 ok 14:53:51 create an oslo.db 15.0.0 release, 14:54:20 which is basically the same as oslo.db 12.3.2 14:54:56 what's the deadline for all this, assuming we want to wait for feedback from stephenfin and TC possibly? 14:55:15 and as frickler said: prepare a release for Caracal *very* *early* and fix services ASAP then 14:55:57 frickler: i'd rather see this sorted out till next week's release meeting 14:56:15 oh, also maybe cut a feature branch from oslo.db before doing reverts so that state is preserved 14:56:35 and then the whole branch can get merged back later 14:56:44 +1 to create a feature branch 14:57:23 speaking of branches: we don't have stable/2023.2 on oslo.db either :-o 14:57:37 nor in castellan 14:57:46 if we don't, then maybe we do not need the reverts 14:58:03 we could branch off an older state instead 14:58:04 sorry, forget what i said 14:58:22 we *have* stable/2023.2 there 14:58:45 (i was on the wrong patch :P) 14:58:55 np 14:59:23 but then the reverts would happen on stable/2023.2 only, right? and master could stay untouched 14:59:25 that' a ton of reverts => https://github.com/openstack/oslo.db/compare/12.3.2..14.0.0?diff=unified 14:59:38 frickler: that sounds good to me 14:59:58 frickler: yeah 15:00:39 but backports would become almost impossible 15:00:57 if a fix is landed on master 15:01:13 well not worse than backports to 2023.1 I guess 15:01:22 yeah 15:01:34 exact 15:02:02 26 patches have landed since 12.3.1 on oslo.db, so yes, that is quite many :/ 15:02:28 what's the amount of works to fix heat? 15:02:34 but that still sounds manageable, better than 100s 15:02:43 though there are revert + trivial doc changes among them 15:02:59 hberaud: IIUC nobody knows exactly yet. but I saw a comment somewhere that sounded bad, let me check 15:03:11 can't the TC force the other patches against services? 15:03:44 https://review.opendev.org/c/openstack/heat/+/886549/comments/37e3a9b7_a4caeb11 15:04:26 the TC could at least put more pressure on those projects where patches are pending 15:04:27 Stephin started his work 2 years ago and raised warning 2 years ago, are we sure that the same situation won't repeat again in 6 months? 15:04:41 s/Stephin/Stephen/ 15:04:43 and maybe do closer tracking of progress 15:05:07 that's my main concern 15:05:09 so maybe ~15 out of 26 patches need to be reverted (or less if we are "lucky"), and hopefully without any conflicts... 15:06:10 because maintained projects are stucks by projects with low maintenance... 15:06:28 gerrit cannot stack reverts automatically I guess, so it will need some manual work. but I think I would volunteer to do that 15:07:06 frickler: thx 15:07:13 oh, can't it? :-o 15:07:26 i thought it handles that well 15:08:15 well I can give it a try 15:08:21 anyway, thanks in advance frickler 15:08:53 so we decide to go ahead with that now and collect feedback on the reviews? 15:09:27 frickler: i'm in favor of this approach 15:09:36 let's go 15:10:12 ok, maybe I won't complete this today, but before Monday evening 15:10:38 np that will lets time to people to react there 15:11:01 agreed: revert oslo.db and castellan patches that causes problems on service side + contact with TC regarding the way forward 15:11:09 is this OK then ^^^ ? 15:11:18 lgtm 15:11:25 hmm, thinking about it a bit, the patches were all on master, so I think gerrit would place the revert onto master, too 15:11:37 so need to do it locally, but that should be fine 15:11:46 Would be good to check with the Octavia folks to make sure they're not using the new Castellan APIs 15:12:07 dmendiza[m]: new castallan isn't in u-c, so I doubt they would 15:12:19 ack 15:12:23 frickler: and what's about services already migrated 15:12:48 all testing is using oslo.db 12.x currently due to u-c 15:12:55 where Stephen's patches are already merged on the service side 15:12:56 and castellan 4.1.0 15:13:02 unless they use directly from master branch, what they shouldn't by default (though it's good to test against master branch in advance) 15:13:11 so they all should work with that, too 15:13:42 as frickler says. they use 12.3.1 15:13:52 so that's good. 15:13:56 After B is released, I'd appreciate *a lot* if we can track all missing patches for SQLA 2.x... 15:14:06 zigo: ++ 15:14:09 (so I can backport them in packages...) 15:14:13 OK, so: 15:14:21 #agreed: revert oslo.db and castellan patches that causes problems on service side + contact with TC regarding the way forward 15:14:26 #agreed revert oslo.db and castellan patches that causes problems on service side + contact with TC regarding the way forward 15:14:48 The current sqla-20 tag isn't good enough, cause it includes what's currently already merged. 15:15:09 i'll add a task to next week to track this on etherpad as well 15:15:21 thx 15:16:03 Cheers. 15:16:30 zigo: we can only have one topic in gerrit, and I don't think changing the current one halfway through is a good idea 15:16:43 so we'd need to do some extra tracking 15:16:53 Agreed, an etherpad is probably better... 15:17:01 or you need to filter what's not in 2023.2 15:17:48 sorry, had to step away for a moment. is there any risk that some services have merged changes depending on things that have merged in newer oslo.db from this cycle? or has constraints saved us from that? 15:18:04 fungi: the latter should hold true 15:18:26 yepp (fingers crossed) 15:18:27 mainly make sure we watch out for oslo.db 15.0.0 breaking services that now need something which got reverted 15:18:42 hopefully won't come up 15:18:50 well the reqs bump should catch that 15:18:55 agreed 15:19:22 at least for anything that's tested on the reqs change anyway 15:19:49 yepp :S 15:20:22 yes, but the number of cross jobs is pretty high. also only unit tests mostly, but we can't check everything 15:20:39 hberaud: as an oslo core member, could you track this task? i can take over other tasks from next week if that helps you 15:20:53 yeah 15:21:23 any special topic to use for the reverts? 15:21:59 "revert-oslo.db-sqla2"? 15:22:05 hberaud: thanks! 15:22:15 "revert-oslo.db-sqla2-bobcat" 15:22:42 and "revert-castellan-bobcat", then? 15:22:47 ++ 15:22:48 yeah 15:23:03 "unf...-everything" ? :) 15:23:10 i wonder what the release notes for those are going to end up looking like 15:23:47 since i don't think reverting release notes actually removes them 15:23:53 well the release notes for oslo.db 15.0.0 would be empty, but we should likely add some explaining 15:24:02 so probably there needs to also be a change merged... yeah that 15:24:09 fungi: we can check it as we did this already in the past :/ 15:24:30 frickler: +1, some explanation would be helpful 15:25:07 anyway, we are quite overtime on the meeting, so let's move on 15:25:14 i think this must be the longest release team meeting in many years 15:25:15 sorry, I need to go grabbing my kids at school 15:25:28 the discussion can be continued after the meeting 15:25:36 hberaud: ack 15:25:42 * hberaud afk 15:26:01 fungi: i think so, too :/ sorry for that 15:26:19 no need to apologize! just an observation 15:26:28 #topic "Assign next week tasks" 15:26:34 all tasks taken 15:26:55 #topic Review countdown email 15:27:06 Yasufumi Ogawa proposed openstack/releases master: Release tacker RC1 for 2023.2 Bobcat https://review.opendev.org/c/openstack/releases/+/894717 15:27:06 #link https://etherpad.opendev.org/p/relmgmt-weekly-emails 15:27:21 please review the mail ^^^ 15:28:30 should that mail mention oslo.db and castellan maybe? 15:28:45 frickler: good point 15:29:11 let me add something about them 15:30:45 * elodilles is searching for the mail thread link 15:34:19 added a dramatic 'Warning' to the mail 15:35:38 OK, let's move on, I'll send the mail some time later today 15:35:53 #topic Open Discussion 15:36:07 any other hot topic? :) 15:37:16 https://review.opendev.org/c/openstack/releases/+/894630 would be nice, but nothing urgent 15:37:51 frickler: ACK, I'll review it :) 15:38:26 OK if nothing else, then 15:38:32 thanks for your patience and for your participation in the meeting! 15:38:37 #endmeeting