Thursday, 2020-07-23

*** ricolin has joined #openstack-release01:13
*** armstrong has joined #openstack-release01:52
*** tinwood has quit IRC02:09
*** tinwood has joined #openstack-release02:11
*** dulek has quit IRC02:42
*** dulek has joined #openstack-release03:09
*** armstrong has quit IRC04:02
*** udesale has joined #openstack-release04:43
*** ykarel has joined #openstack-release05:04
*** vishalmanchanda has joined #openstack-release05:39
openstackgerritzhaoleilc proposed openstack/releases master: Correct a typo of an annotation
*** jbadiapa has joined #openstack-release06:54
*** tinwood is now known as tinwood-afk06:55
*** amoralej|off is now known as amoralej07:17
*** tosky has joined #openstack-release07:22
*** elod_off is now known as elod07:54
*** e0ne has joined #openstack-release08:24
*** ykarel has quit IRC08:32
*** tinwood-afk is now known as tinwood08:34
*** ykarel_ has joined #openstack-release08:35
openstackgerritMerged openstack/releases master: Oslo releases for 2020-07-22
*** ykarel_ is now known as ykarel09:02
*** vishalmanchanda has quit IRC09:10
openstackgerritMerged openstack/releases master: Release cinderlib 2.0.0
openstackgerritMerged openstack/releases master: Release octavia-dashboard for stable/stein
openstackgerritMerged openstack/releases master: Release neutron for stable/stein
openstackgerritMerged openstack/releases master: Release magnum for stable/stein
openstackgerritMerged openstack/releases master: Release designate for stable/stein
openstackgerritMerged openstack/releases master: Release barbican for stable/stein
*** ricolin has quit IRC10:13
*** dtantsur|afk is now known as dtantsur10:29
*** udesale_ has joined #openstack-release11:03
*** udesale has quit IRC11:06
*** vishalmanchanda has joined #openstack-release11:09
*** ricolin has joined #openstack-release11:37
ttxfungi: recent issues with AFS publication. See for summary12:09
fungiwill look12:13
*** ricolin has quit IRC12:13
fungiso this is the part that's failing:
fungiit looks like for some reason afs_target was /afs/ so missing the project name?12:18
fungiand the tarballs path and all that12:18
fungii'll see if there were recent changes to how afs_target is getting populated12:19
*** amoralej is now known as amoralej|lunch12:25
fungii don't get it12:26
fungieven if the project name were unset for some reason, /afs/ is what it would be trying to create12:27
fungier, was what i meant to link there12:36
fungihrm, maybe that error is misleading, says it should have tried to create /afs/
*** ricolin has joined #openstack-release12:38
fungimaybe we have an executor with broken afs12:38
fungiis this happening consistently or just occasionally?12:39
fungicoincidentally all the tarball archive failures happened on ze1112:40
fungithe opaque docs failure happened on ze10 though12:41
fungii'll check them both out12:41
rpittauhi all, we've just noticed that in the ironic-python-agent release note page we have version 6.1.0 mentioned in both Ussuri and Current sections, doesn't really look right :/13:00
smcginnisrpittau: That should go away once victoria is released I think.13:05
rpittausmcginnis: ok, thanks13:06
smcginnisrpittau: Though looking closer, this is a cycle-with-intermediary deliverable. We typically only have the diff-start setting for cycle-with-rc deliverables.13:08
fungiupdate on the release job failures from earlier today, it looks like ze02, ze10 and ze11 unexpectedly rebooted within the past 24 hours, so i'm looking to see if that could be related13:08
smcginnisrpittau: So maybe something needs to be set there. I haven't seen this before. I don't have time to dig in right now, but I can try to look later if you can't find the reason.13:09
*** ykarel is now known as ykarel|away13:23
*** amoralej|lunch is now known as amoralej13:26
*** ykarel|away has quit IRC13:28
rpittausmcginnis: thanks! I was trying to understand but kind of hit a wall, I'll keep looking, if you can help at some point I would really appreciate that :)13:30
openstackgerritRiccardo Pittau proposed openstack/releases master: Release ironic-python-agent 6.1.1 for ussuri
*** ykarel|away has joined #openstack-release15:32
*** armstrong has joined #openstack-release15:34
*** udesale_ has quit IRC15:42
*** ykarel_ has joined #openstack-release15:50
*** ykarel|away has quit IRC15:53
smcginnis#startmeeting releaseteam16:00
openstackMeeting started Thu Jul 23 16:00:01 2020 UTC and is due to finish in 60 minutes.  The chair is smcginnis. Information about MeetBot at
openstackUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.16:00
*** openstack changes topic to " (Meeting topic: releaseteam)"16:00
openstackThe meeting name has been set to 'releaseteam'16:00
smcginnisPing list: ttx armstrong16:00
smcginnis#link Agenda16:00
smcginnisLine 178-ish16:00
smcginnis203 for the meeting agenda, if you want to be technical.16:00
smcginnis#topic Review tasks completion16:01
*** openstack changes topic to "Review tasks completion (Meeting topic: releaseteam)"16:01
smcginnisttx: Want to cover the governance checks?16:01
ttxSo that check basically checks if we have the deliverables files matcging current state of governance16:02
ttxThere were a few extras, which I removed at
ttxThere are also a few missing, which we need to review16:03
ttxDeliverable files are missing when new deliverables get added and no deliverable file is created16:03
ttxLike for example oslo.metrics, added May 616:04
ttxFor those, the question is, are they going to be ready for a victoria release16:04
smcginnisIs that process documented somewhere that we can call out the step of adding the new deliverable file? I can't recall at the moment.16:04
ttxIt does not make sense until the deliverable is ready to release16:04
ttxwhich can take short or long timew16:04
ttxbetter to review it as consistency check16:05
smcginnisTwo added since this May. Those I could see not being ready, but the others listed have had some time.16:05
smcginnisYeah, true.16:05
ttxThe ones added in May+ i would not even look into. If ready they will let us know, otherwise we'll pick then up next time16:05
ttxthat leaves us with:16:06
ttxmonasca-ceilometer and monasca-log-api ... were released in train, but not in ussuri16:06
ttxWe need to ask monasca folks what their plans are16:06
smcginnisRepos are not retired.16:06
ttxis it abandoned? If not, why did we skip them in ussuri?16:06
ttxand should we track them for victoria ?16:07
smcginnisAh, they were marked as deprecated:
ttxhmm, not in governance :)16:07
smcginnisSo at least those two, I think we should remove the files. They should follow up on governance updates.16:07
ttxso maybe the fix here is to mark them as deprecated in governance16:07
smcginnisI think so.16:08
ttxbarbican-ui (Added Oct 2019) -- never released yet16:08
ttxwould be good to ask for their plans16:09
ttxjs-openstack-lib (Added January 9) -- never released yet16:09
smcginnisMaybe mordred would know?16:09
ttxthe bunch of xstatic things... I don;t see what their point is if they don;t get released16:09
smcginnise0ne: ^16:10
ttxyes all of those are tasks that we need to follow up with people16:10
fungii thought they did get released, but they needed "special" version numbering?16:10
ttxfungi: yes. My point is...16:10
smcginnisDefinitely some of the xstatic ones have been released.16:10
ttxTheir only point is to be released16:10
ttxthere is no "work" in them, just a packaging shell for Pypi release16:11
fungii see, some were added and never released16:11
ttxso I'm surprised why they would be created but not released yet16:11
ttxfinally openstack-tempest-skiplist (Added Mar 20)16:11
ttxno idea if the plan was to release that16:11
ttxLast two I would ignore for now as too young16:12
ttxWho can do the followup? I'm off next week so would rather not take it16:12
smcginnisI can call them out in the countdown email at least. I didn't do all of them in this week's (that I just finally sent out yesterday).16:13
smcginnisOr maybe better if I do it as its own message.16:13
smcginnisThat way it might get more visibility and I can tag affected projects.16:13
ttxI would try to ping people in IRC, but your call :)16:14
smcginnisIf someone can do that, it would be best. I'm not sure if I will have time to, but I can try.16:14
ttxnot super urgent16:14
ttxWe can pick it up at next meeting if we prefer16:14
smcginnisLet's see how far we can get.16:15
smcginnisThe other task is the countdown, and I have written down a big reminder to make sure I don't get too busy and forget to send it tomorrow.16:15
smcginnis#topic Octavia EOL releases16:16
*** openstack changes topic to "Octavia EOL releases (Meeting topic: releaseteam)"16:16
smcginnisYeah, I think those are ready.16:16
smcginnisThere are a couple Cinder ones now too.16:16
smcginnisWe would just need to follow up removing the branches.16:16
ttxI was unclear if there were ok to +2a16:16
ttxwill do now16:16
smcginnisI forget, did we figure out the release managers have the necessary permissions to delete those branches?16:17
smcginnisI know we talked about it, I just can't remember what we determined.16:17
ttxwe can't delete branches16:17
ttxonly create them.16:17
smcginnisOK. At least we can bug fungi :)16:17
fungiyep, also i can temporarily grant that permission16:18
fungiit's just that under the version of gerrit we're still on, that permission comes lumped in with a bunch of much more dangerous ones16:18
fungiso even my admin account doesn't have that granted to it normally16:18
smcginnisI'd rather not have the rights to be more dangerous. ;)16:19
smcginnisWe can follow up on those afterwards.16:19
fungismcginnis 007, license to delete16:19
elodhi, sorry, just one question: is there an easy way / common place to check which repositories are EOL'd on a certain branch?16:19
smcginnisThey should have a $series-eol tag.16:20
ttxfungi: re your earlier question about job fails, about 15% of the jobs had an AFS failure this morning16:20
smcginnisBut not a great visible way like we had with the table in releases.o.o.16:20
ttxthe others worked ok16:20
smcginnis#topic Review email content16:20
*** openstack changes topic to "Review email content (Meeting topic: releaseteam)"16:20
elodsmcginnis: ok, thanks16:21
smcginnisNothing too exciting. Just reminders of the upcoming deadlines.16:21
fungittx: thanks, that does lead me to believe it could either be a temporary connectivity problem or an issue with afs writes from a subset of our executors16:21
ttxthat sounds off16:21
fungii'm hoping to get back to running those errors down the rest of the way shortly16:21
smcginnisttx: Yeah, too soon.16:21
ttxsmcginnis: Victoria-2 is next week.16:21
smcginnisIs this a skip week?16:21
* smcginnis looks again16:21
ttxso the email you sent this week is the right one16:21
ttxjust a bit early, rather than a bit late16:21
armstrongfungi: does the permissions on Repos come from the Infra team?16:22
smcginnisHonestly, really not liking having the emails in the process docs. It's a bit confusing.16:22
ttxIt should raelly not be confusing. Just tells you what to send every week :)16:22
smcginnisIt shouldn't be, but it has been.16:22
ttxShoudl probably say " at the end of the week, send16:23
smcginnisI was thinking about some sort of script to use schedule.yaml and jinja templates, but that's probably overkill. :)16:23
ttxOK so the email for this week was sent already16:23
fungiarmstrong: there are access controls which we manage in a git repository, but those also inherit from a shared access configuration where we centrally grant some permissions to specific gerrit groups. one group called "project bootstrappers" is used by our project creation automation and has basically full access to delete things from a repository, so one of our admins generally adds themselves or16:24
fungisome delegate to that group temporarily to do things like branch deletion16:24
ttx(there was none to send last week)16:24
smcginnisLet's move on then.16:24
smcginnis#topic AFS-related job failures16:24
*** openstack changes topic to "AFS-related job failures (Meeting topic: releaseteam)"16:24
fungiyeah, i'm looking into these16:25
*** amoralej is now known as amoralej|off16:25
smcginnisPotentially something to do with nodes restarting?16:25
ttxThose will liekly require manual work to fix16:25
fungiso far it appears that the two tarball write issues happened from ze11, and the rather opaque docs upload error came from a build on ze1016:25
smcginnisSo the important ones are probably oslo.messaging and designate.16:26
fungiand i noticed that both of those executors spontaneously rebooted (perhaps provider was doing a reboot migration to another host) in the past day, though still hours before the failed builds16:26
smcginnisThey were tagged but no tarballs uploaded?16:26
ttxthe pypi uploads worked, so we need to find a way to upload the corresponding tarballs16:26
fungiyeah, i can do that part manually16:26
ttx+ Missing constraint updates16:26
fungiand also the signatures, now that we actually spit out copies of them in the job logs16:26
ttxmissing release announce we can probably survive16:26
fungii need to test afs writes from ze11 and also check the executor debug log from ze10 to see what specifically the docs error was16:27
smcginnisThese were stable releases, so the nightly constraints update won't pick up oslo.messaging.16:27
smcginnisI can propose that one.16:27
fungialso all three failures occurred within an hour of each other, so it's possible this was a short-lived network connectivity issue16:27
ttxAFS appears to be exceptionally brittle, or at least not liking our setup :)16:27
fungiwell, if it was a connectivity issue, scp and rsync would have broken similarly16:28
smcginnisAh, oslo.messaging one was victoria, so that actually will be picked up by the nightly updates.16:29
smcginnisSo we just need the tarballs.16:29
smcginnisfungi: Seems like scp and rsync have more retrying built in though.16:29
*** diablo_rojo_phon has joined #openstack-release16:29
*** ricolin has quit IRC16:30
smcginnisfungi: I've put a note in the etherpad as a reminder that we will need you to upload the tarballs. That good?16:30
fungiafs actually does too16:30
fungiyep, that's good16:31
smcginnis#topic Assign tasks for R-11 week16:31
*** openstack changes topic to "Assign tasks for R-11 week (Meeting topic: releaseteam)"16:31
smcginnisttx is out all week, so unfortunately we can't assign them all to him.16:31
fungi(it's the command line tools which aren't retrying because they treat afs like they do a local filesystem, afs itself continually rechecks for connectivity to be reestablished)16:31
ttxI mean, you /can/16:31
smcginnisMaybe hberaud would be willing to pick those up.16:32
smcginnisI will leave it unassigned for now and do them if no one else can.16:32
smcginnisMostly just running some scripts and then seeing if there is anything to do based on that.16:33
smcginnis#topic AOB16:33
*** openstack changes topic to "AOB (Meeting topic: releaseteam)"16:33
smcginnisAnything else?16:34
openstackgerritMerged openstack/releases master: Octavia: EOL Rocky
openstackgerritMerged openstack/releases master: Octavia: EOL Queens branch
smcginnisOK, we can end early then. \o/16:34
smcginnisThanks everyone.16:34
*** openstack changes topic to "OpenStack Release Managers office - Come here to discuss how to release OpenStack components - Logged at"16:35
openstackMeeting ended Thu Jul 23 16:35:07 2020 UTC.  Information about MeetBot at . (v 0.1.4)16:35
openstackMinutes (text):
*** dtantsur is now known as dtantsur|afk16:37
openstackgerritSean McGinnis proposed openstack/releases master: Update Summit to be virtual event
openstackgerritLee Yarwood proposed openstack/releases master: nova: Release 19.2.1
*** ykarel_ has quit IRC16:50
openstackgerritElod Illes proposed openstack/releases master: nova: Release 19.2.1
*** markmcclain has quit IRC17:14
*** markmcclain has joined #openstack-release17:16
*** vishalmanchanda has quit IRC17:28
fungi#status log copied oslo.messaging 12.2.2 and designate 8.0.1 wheels/sdists from pypi to the tarballs site after verifying with release key signatures (also uploaded) from build failure logs18:38
openstackstatusfungi: finished logging18:38
fungittx: smcginnis: ^18:38
fungii also did it from ze11, where those builds originally failed, and did not get any errors18:38
fungiso it must have been a transient issue18:39
smcginnisAwesome, thanks fungi.18:40
fungino sweat18:41
*** armstrong has quit IRC18:44
openstackgerritwes hayutin proposed openstack/releases master: minor update release for tripleo ussuri
openstackgerritwes hayutin proposed openstack/releases master: minor update release for tripleo ussuri
e0nettx, smcginnis: hi. AFAIK, we don't need new xstatic releases because we don't upgrade horizon dependencies. I'll recheck tomorrow20:14
smcginnisThanks e0ne. I will put up a patch to remove those deliverable files. After you check on things tomorrow, please +1 that if we are good to go ahead with removing them.20:32
e0nesmcginnis: ok, thanks for watching for xstatic-* releases20:34
openstackgerritwes hayutin proposed openstack/releases master: tripleo victoria-1 release
*** e0ne has quit IRC20:41
*** dave-mccowan has joined #openstack-release20:41
*** jbadiapa has quit IRC22:55
*** tosky has quit IRC22:59

Generated by 2.17.2 by Marius Gedminas - find it at!