| *** ricolin has joined #openstack-release | 01:13 | |
| *** armstrong has joined #openstack-release | 01:52 | |
| *** tinwood has quit IRC | 02:09 | |
| *** tinwood has joined #openstack-release | 02:11 | |
| *** dulek has quit IRC | 02:42 | |
| *** dulek has joined #openstack-release | 03:09 | |
| *** armstrong has quit IRC | 04:02 | |
| *** udesale has joined #openstack-release | 04:43 | |
| *** ykarel has joined #openstack-release | 05:04 | |
| *** vishalmanchanda has joined #openstack-release | 05:39 | |
| openstackgerrit | zhaoleilc proposed openstack/releases master: Correct a typo of an annotation https://review.opendev.org/742597 | 06:26 |
|---|---|---|
| *** jbadiapa has joined #openstack-release | 06:54 | |
| *** tinwood is now known as tinwood-afk | 06:55 | |
| *** amoralej|off is now known as amoralej | 07:17 | |
| *** tosky has joined #openstack-release | 07:22 | |
| *** elod_off is now known as elod | 07:54 | |
| *** e0ne has joined #openstack-release | 08:24 | |
| *** ykarel has quit IRC | 08:32 | |
| *** tinwood-afk is now known as tinwood | 08:34 | |
| *** ykarel_ has joined #openstack-release | 08:35 | |
| openstackgerrit | Merged openstack/releases master: Oslo releases for 2020-07-22 https://review.opendev.org/742484 | 08:55 |
| *** ykarel_ is now known as ykarel | 09:02 | |
| *** vishalmanchanda has quit IRC | 09:10 | |
| openstackgerrit | Merged openstack/releases master: Release cinderlib 2.0.0 https://review.opendev.org/742503 | 09:16 |
| openstackgerrit | Merged openstack/releases master: Release octavia-dashboard for stable/stein https://review.opendev.org/741761 | 09:16 |
| openstackgerrit | Merged openstack/releases master: Release neutron for stable/stein https://review.opendev.org/741758 | 09:16 |
| openstackgerrit | Merged openstack/releases master: Release magnum for stable/stein https://review.opendev.org/741745 | 09:16 |
| openstackgerrit | Merged openstack/releases master: Release designate for stable/stein https://review.opendev.org/741735 | 09:16 |
| openstackgerrit | Merged openstack/releases master: Release barbican for stable/stein https://review.opendev.org/741728 | 09:16 |
| *** ricolin has quit IRC | 10:13 | |
| *** dtantsur|afk is now known as dtantsur | 10:29 | |
| *** udesale_ has joined #openstack-release | 11:03 | |
| *** udesale has quit IRC | 11:06 | |
| *** vishalmanchanda has joined #openstack-release | 11:09 | |
| *** ricolin has joined #openstack-release | 11:37 | |
| ttx | fungi: recent issues with AFS publication. See http://lists.openstack.org/pipermail/openstack-discuss/2020-July/016064.html for summary | 12:09 |
| fungi | will look | 12:13 |
| *** ricolin has quit IRC | 12:13 | |
| fungi | so this is the part that's failing: https://opendev.org/zuul/zuul-jobs/src/branch/master/roles/upload-afs-synchronize/tasks/main.yaml#L4-L6 | 12:17 |
| fungi | it looks like for some reason afs_target was /afs/.openstack.org so missing the project name? | 12:18 |
| fungi | and the tarballs path and all that | 12:18 |
| fungi | i'll see if there were recent changes to how afs_target is getting populated | 12:19 |
| *** amoralej is now known as amoralej|lunch | 12:25 | |
| fungi | zuul/zuul-jobs/src/branch/master/roles/upload-afs-synchronize/tasks/main.yaml#L4-L6 | 12:26 |
| fungi | i don't get it | 12:26 |
| fungi | even if the project name were unset for some reason, /afs/.openstack.org/project/tarballs.opendev.org/ is what it would be trying to create | 12:27 |
| fungi | er, https://opendev.org/openstack/project-config/src/branch/master/playbooks/publish/openstack-artifacts.yaml#L7-L12 was what i meant to link there | 12:36 |
| fungi | hrm, maybe that error is misleading, https://zuul.opendev.org/t/openstack/build/dfce7b106189491b9d9026a079c06bdd/console#5/0/7/localhost says it should have tried to create /afs/.openstack.org/project/tarballs.opendev.org/openstack/oslo.messaging/ | 12:38 |
| *** ricolin has joined #openstack-release | 12:38 | |
| fungi | maybe we have an executor with broken afs | 12:38 |
| fungi | is this happening consistently or just occasionally? | 12:39 |
| fungi | coincidentally all the tarball archive failures happened on ze11 | 12:40 |
| fungi | the opaque docs failure happened on ze10 though | 12:41 |
| fungi | i'll check them both out | 12:41 |
| rpittau | hi all, we've just noticed that in the ironic-python-agent release note page we have version 6.1.0 mentioned in both Ussuri and Current sections, doesn't really look right :/ | 13:00 |
| smcginnis | rpittau: That should go away once victoria is released I think. | 13:05 |
| rpittau | smcginnis: ok, thanks | 13:06 |
| smcginnis | rpittau: Though looking closer, this is a cycle-with-intermediary deliverable. We typically only have the diff-start setting for cycle-with-rc deliverables. | 13:08 |
| fungi | update on the release job failures from earlier today, it looks like ze02, ze10 and ze11 unexpectedly rebooted within the past 24 hours, so i'm looking to see if that could be related | 13:08 |
| smcginnis | rpittau: So maybe something needs to be set there. I haven't seen this before. I don't have time to dig in right now, but I can try to look later if you can't find the reason. | 13:09 |
| *** ykarel is now known as ykarel|away | 13:23 | |
| *** amoralej|lunch is now known as amoralej | 13:26 | |
| *** ykarel|away has quit IRC | 13:28 | |
| rpittau | smcginnis: thanks! I was trying to understand but kind of hit a wall, I'll keep looking, if you can help at some point I would really appreciate that :) | 13:30 |
| openstackgerrit | Riccardo Pittau proposed openstack/releases master: Release ironic-python-agent 6.1.1 for ussuri https://review.opendev.org/742680 | 14:10 |
| *** ykarel|away has joined #openstack-release | 15:32 | |
| *** armstrong has joined #openstack-release | 15:34 | |
| *** udesale_ has quit IRC | 15:42 | |
| *** ykarel_ has joined #openstack-release | 15:50 | |
| *** ykarel|away has quit IRC | 15:53 | |
| smcginnis | #startmeeting releaseteam | 16:00 |
| openstack | Meeting started Thu Jul 23 16:00:01 2020 UTC and is due to finish in 60 minutes. The chair is smcginnis. Information about MeetBot at http://wiki.debian.org/MeetBot. | 16:00 |
| openstack | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 16:00 |
| *** openstack changes topic to " (Meeting topic: releaseteam)" | 16:00 | |
| openstack | The meeting name has been set to 'releaseteam' | 16:00 |
| smcginnis | Ping list: ttx armstrong | 16:00 |
| ttx | o/ | 16:00 |
| smcginnis | #link https://etherpad.opendev.org/p/victoria-relmgt-tracking Agenda | 16:00 |
| armstrong | o/ | 16:00 |
| smcginnis | Line 178-ish | 16:00 |
| smcginnis | 203 for the meeting agenda, if you want to be technical. | 16:00 |
| smcginnis | #topic Review tasks completion | 16:01 |
| *** openstack changes topic to "Review tasks completion (Meeting topic: releaseteam)" | 16:01 | |
| smcginnis | ttx: Want to cover the governance checks? | 16:01 |
| ttx | sure | 16:01 |
| ttx | So that check basically checks if we have the deliverables files matcging current state of governance | 16:02 |
| ttx | There were a few extras, which I removed at https://review.opendev.org/742200 | 16:03 |
| ttx | There are also a few missing, which we need to review | 16:03 |
| ttx | Deliverable files are missing when new deliverables get added and no deliverable file is created | 16:03 |
| ttx | Like for example oslo.metrics, added May 6 | 16:04 |
| ttx | For those, the question is, are they going to be ready for a victoria release | 16:04 |
| smcginnis | Is that process documented somewhere that we can call out the step of adding the new deliverable file? I can't recall at the moment. | 16:04 |
| ttx | It does not make sense until the deliverable is ready to release | 16:04 |
| ttx | which can take short or long timew | 16:04 |
| ttx | better to review it as consistency check | 16:05 |
| smcginnis | Two added since this May. Those I could see not being ready, but the others listed have had some time. | 16:05 |
| smcginnis | Yeah, true. | 16:05 |
| ttx | The ones added in May+ i would not even look into. If ready they will let us know, otherwise we'll pick then up next time | 16:05 |
| ttx | that leaves us with: | 16:06 |
| ttx | monasca-ceilometer and monasca-log-api ... were released in train, but not in ussuri | 16:06 |
| ttx | We need to ask monasca folks what their plans are | 16:06 |
| smcginnis | Repos are not retired. | 16:06 |
| ttx | is it abandoned? If not, why did we skip them in ussuri? | 16:06 |
| ttx | and should we track them for victoria ? | 16:07 |
| smcginnis | Ah, they were marked as deprecated: https://opendev.org/openstack/monasca-ceilometer/commit/875bc660ee68664d0ab4a21442c69ffd164d2ddf | 16:07 |
| ttx | hmm, not in governance :) | 16:07 |
| smcginnis | And https://opendev.org/openstack/monasca-log-api/commit/4eccad156f282f2eb300be7a306703c90dcba996 | 16:07 |
| smcginnis | So at least those two, I think we should remove the files. They should follow up on governance updates. | 16:07 |
| ttx | so maybe the fix here is to mark them as deprecated in governance | 16:07 |
| smcginnis | I think so. | 16:08 |
| ttx | barbican-ui (Added Oct 2019) -- never released yet | 16:08 |
| ttx | would be good to ask for their plans | 16:09 |
| ttx | js-openstack-lib (Added January 9) -- never released yet | 16:09 |
| smcginnis | Maybe mordred would know? | 16:09 |
| ttx | the bunch of xstatic things... I don;t see what their point is if they don;t get released | 16:09 |
| smcginnis | e0ne: ^ | 16:10 |
| ttx | yes all of those are tasks that we need to follow up with people | 16:10 |
| fungi | i thought they did get released, but they needed "special" version numbering? | 16:10 |
| ttx | fungi: yes. My point is... | 16:10 |
| smcginnis | Definitely some of the xstatic ones have been released. | 16:10 |
| ttx | Their only point is to be released | 16:10 |
| fungi | true | 16:10 |
| ttx | there is no "work" in them, just a packaging shell for Pypi release | 16:11 |
| fungi | i see, some were added and never released | 16:11 |
| ttx | so I'm surprised why they would be created but not released yet | 16:11 |
| ttx | finally openstack-tempest-skiplist (Added Mar 20) | 16:11 |
| ttx | no idea if the plan was to release that | 16:11 |
| ttx | Last two I would ignore for now as too young | 16:12 |
| ttx | Who can do the followup? I'm off next week so would rather not take it | 16:12 |
| smcginnis | I can call them out in the countdown email at least. I didn't do all of them in this week's (that I just finally sent out yesterday). | 16:13 |
| smcginnis | Or maybe better if I do it as its own message. | 16:13 |
| smcginnis | That way it might get more visibility and I can tag affected projects. | 16:13 |
| ttx | I would try to ping people in IRC, but your call :) | 16:14 |
| smcginnis | If someone can do that, it would be best. I'm not sure if I will have time to, but I can try. | 16:14 |
| ttx | not super urgent | 16:14 |
| ttx | We can pick it up at next meeting if we prefer | 16:14 |
| smcginnis | Let's see how far we can get. | 16:15 |
| smcginnis | The other task is the countdown, and I have written down a big reminder to make sure I don't get too busy and forget to send it tomorrow. | 16:15 |
| smcginnis | #topic Octavia EOL releases | 16:16 |
| *** openstack changes topic to "Octavia EOL releases (Meeting topic: releaseteam)" | 16:16 | |
| smcginnis | #link https://review.opendev.org/#/c/741272/ | 16:16 |
| smcginnis | #link https://review.opendev.org/#/c/719099/ | 16:16 |
| smcginnis | Yeah, I think those are ready. | 16:16 |
| smcginnis | There are a couple Cinder ones now too. | 16:16 |
| smcginnis | We would just need to follow up removing the branches. | 16:16 |
| ttx | I was unclear if there were ok to +2a | 16:16 |
| ttx | will do now | 16:16 |
| smcginnis | I forget, did we figure out the release managers have the necessary permissions to delete those branches? | 16:17 |
| smcginnis | I know we talked about it, I just can't remember what we determined. | 16:17 |
| ttx | we can't delete branches | 16:17 |
| ttx | only create them. | 16:17 |
| smcginnis | OK. At least we can bug fungi :) | 16:17 |
| fungi | yep, also i can temporarily grant that permission | 16:18 |
| fungi | it's just that under the version of gerrit we're still on, that permission comes lumped in with a bunch of much more dangerous ones | 16:18 |
| fungi | so even my admin account doesn't have that granted to it normally | 16:18 |
| smcginnis | I'd rather not have the rights to be more dangerous. ;) | 16:19 |
| smcginnis | We can follow up on those afterwards. | 16:19 |
| fungi | smcginnis 007, license to delete | 16:19 |
| elod | hi, sorry, just one question: is there an easy way / common place to check which repositories are EOL'd on a certain branch? | 16:19 |
| smcginnis | They should have a $series-eol tag. | 16:20 |
| ttx | fungi: re your earlier question about job fails, about 15% of the jobs had an AFS failure this morning | 16:20 |
| smcginnis | But not a great visible way like we had with the table in releases.o.o. | 16:20 |
| ttx | the others worked ok | 16:20 |
| smcginnis | #topic Review email content | 16:20 |
| *** openstack changes topic to "Review email content (Meeting topic: releaseteam)" | 16:20 | |
| smcginnis | #link https://etherpad.opendev.org/p/relmgmt-weekly-emails | 16:20 |
| elod | smcginnis: ok, thanks | 16:21 |
| smcginnis | Nothing too exciting. Just reminders of the upcoming deadlines. | 16:21 |
| ttx | hhhm | 16:21 |
| fungi | ttx: thanks, that does lead me to believe it could either be a temporary connectivity problem or an issue with afs writes from a subset of our executors | 16:21 |
| ttx | that sounds off | 16:21 |
| fungi | i'm hoping to get back to running those errors down the rest of the way shortly | 16:21 |
| smcginnis | ttx: Yeah, too soon. | 16:21 |
| ttx | smcginnis: Victoria-2 is next week. | 16:21 |
| smcginnis | Is this a skip week? | 16:21 |
| * smcginnis looks again | 16:21 | |
| ttx | so the email you sent this week is the right one | 16:21 |
| ttx | just a bit early, rather than a bit late | 16:21 |
| armstrong | fungi: does the permissions on Repos come from the Infra team? | 16:22 |
| smcginnis | Honestly, really not liking having the emails in the process docs. It's a bit confusing. | 16:22 |
| ttx | It should raelly not be confusing. Just tells you what to send every week :) | 16:22 |
| smcginnis | It shouldn't be, but it has been. | 16:22 |
| ttx | Shoudl probably say " at the end of the week, send | 16:23 |
| smcginnis | I was thinking about some sort of script to use schedule.yaml and jinja templates, but that's probably overkill. :) | 16:23 |
| ttx | OK so the email for this week was sent already | 16:23 |
| smcginnis | ☑️ | 16:24 |
| fungi | armstrong: there are access controls which we manage in a git repository, but those also inherit from a shared access configuration where we centrally grant some permissions to specific gerrit groups. one group called "project bootstrappers" is used by our project creation automation and has basically full access to delete things from a repository, so one of our admins generally adds themselves or | 16:24 |
| fungi | some delegate to that group temporarily to do things like branch deletion | 16:24 |
| ttx | (there was none to send last week) | 16:24 |
| smcginnis | Let's move on then. | 16:24 |
| smcginnis | #topic AFS-related job failures | 16:24 |
| *** openstack changes topic to "AFS-related job failures (Meeting topic: releaseteam)" | 16:24 | |
| smcginnis | #link http://lists.openstack.org/pipermail/openstack-discuss/2020-July/016064.html | 16:24 |
| fungi | yeah, i'm looking into these | 16:25 |
| *** amoralej is now known as amoralej|off | 16:25 | |
| smcginnis | Potentially something to do with nodes restarting? | 16:25 |
| ttx | Those will liekly require manual work to fix | 16:25 |
| fungi | so far it appears that the two tarball write issues happened from ze11, and the rather opaque docs upload error came from a build on ze10 | 16:25 |
| smcginnis | So the important ones are probably oslo.messaging and designate. | 16:26 |
| fungi | and i noticed that both of those executors spontaneously rebooted (perhaps provider was doing a reboot migration to another host) in the past day, though still hours before the failed builds | 16:26 |
| smcginnis | They were tagged but no tarballs uploaded? | 16:26 |
| ttx | the pypi uploads worked, so we need to find a way to upload the corresponding tarballs | 16:26 |
| fungi | yeah, i can do that part manually | 16:26 |
| ttx | + Missing constraint updates | 16:26 |
| fungi | and also the signatures, now that we actually spit out copies of them in the job logs | 16:26 |
| ttx | missing release announce we can probably survive | 16:26 |
| fungi | i need to test afs writes from ze11 and also check the executor debug log from ze10 to see what specifically the docs error was | 16:27 |
| smcginnis | These were stable releases, so the nightly constraints update won't pick up oslo.messaging. | 16:27 |
| smcginnis | I can propose that one. | 16:27 |
| fungi | also all three failures occurred within an hour of each other, so it's possible this was a short-lived network connectivity issue | 16:27 |
| ttx | AFS appears to be exceptionally brittle, or at least not liking our setup :) | 16:27 |
| fungi | well, if it was a connectivity issue, scp and rsync would have broken similarly | 16:28 |
| smcginnis | Ah, oslo.messaging one was victoria, so that actually will be picked up by the nightly updates. | 16:29 |
| smcginnis | So we just need the tarballs. | 16:29 |
| smcginnis | fungi: Seems like scp and rsync have more retrying built in though. | 16:29 |
| *** diablo_rojo_phon has joined #openstack-release | 16:29 | |
| *** ricolin has quit IRC | 16:30 | |
| smcginnis | fungi: I've put a note in the etherpad as a reminder that we will need you to upload the tarballs. That good? | 16:30 |
| fungi | afs actually does too | 16:30 |
| fungi | yep, that's good | 16:31 |
| smcginnis | Thanks! | 16:31 |
| smcginnis | #topic Assign tasks for R-11 week | 16:31 |
| *** openstack changes topic to "Assign tasks for R-11 week (Meeting topic: releaseteam)" | 16:31 | |
| smcginnis | ttx is out all week, so unfortunately we can't assign them all to him. | 16:31 |
| fungi | (it's the command line tools which aren't retrying because they treat afs like they do a local filesystem, afs itself continually rechecks for connectivity to be reestablished) | 16:31 |
| ttx | I mean, you /can/ | 16:31 |
| smcginnis | :) | 16:32 |
| smcginnis | Maybe hberaud would be willing to pick those up. | 16:32 |
| smcginnis | I will leave it unassigned for now and do them if no one else can. | 16:32 |
| smcginnis | Mostly just running some scripts and then seeing if there is anything to do based on that. | 16:33 |
| ttx | ++ | 16:33 |
| smcginnis | #topic AOB | 16:33 |
| *** openstack changes topic to "AOB (Meeting topic: releaseteam)" | 16:33 | |
| smcginnis | Anything else? | 16:34 |
| openstackgerrit | Merged openstack/releases master: Octavia: EOL Rocky https://review.opendev.org/741272 | 16:34 |
| openstackgerrit | Merged openstack/releases master: Octavia: EOL Queens branch https://review.opendev.org/719099 | 16:34 |
| smcginnis | OK, we can end early then. \o/ | 16:34 |
| smcginnis | Thanks everyone. | 16:34 |
| ttx | o/ | 16:34 |
| ttx | Thanks | 16:34 |
| smcginnis | #endmeeting | 16:35 |
| *** openstack changes topic to "OpenStack Release Managers office - Come here to discuss how to release OpenStack components - Logged at http://eavesdrop.openstack.org/irclogs/%23openstack-release/" | 16:35 | |
| openstack | Meeting ended Thu Jul 23 16:35:07 2020 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 16:35 |
| openstack | Minutes: http://eavesdrop.openstack.org/meetings/releaseteam/2020/releaseteam.2020-07-23-16.00.html | 16:35 |
| openstack | Minutes (text): http://eavesdrop.openstack.org/meetings/releaseteam/2020/releaseteam.2020-07-23-16.00.txt | 16:35 |
| openstack | Log: http://eavesdrop.openstack.org/meetings/releaseteam/2020/releaseteam.2020-07-23-16.00.log.html | 16:35 |
| *** dtantsur is now known as dtantsur|afk | 16:37 | |
| openstackgerrit | Sean McGinnis proposed openstack/releases master: Update Summit to be virtual event https://review.opendev.org/742705 | 16:37 |
| openstackgerrit | Lee Yarwood proposed openstack/releases master: nova: Release 19.2.1 https://review.opendev.org/741760 | 16:38 |
| *** ykarel_ has quit IRC | 16:50 | |
| openstackgerrit | Elod Illes proposed openstack/releases master: nova: Release 19.2.1 https://review.opendev.org/741760 | 16:54 |
| *** markmcclain has quit IRC | 17:14 | |
| *** markmcclain has joined #openstack-release | 17:16 | |
| *** vishalmanchanda has quit IRC | 17:28 | |
| fungi | #status log copied oslo.messaging 12.2.2 and designate 8.0.1 wheels/sdists from pypi to the tarballs site after verifying with release key signatures (also uploaded) from build failure logs | 18:38 |
| openstackstatus | fungi: finished logging | 18:38 |
| fungi | ttx: smcginnis: ^ | 18:38 |
| fungi | i also did it from ze11, where those builds originally failed, and did not get any errors | 18:38 |
| fungi | so it must have been a transient issue | 18:39 |
| smcginnis | Awesome, thanks fungi. | 18:40 |
| fungi | no sweat | 18:41 |
| *** armstrong has quit IRC | 18:44 | |
| openstackgerrit | wes hayutin proposed openstack/releases master: minor update release for tripleo ussuri https://review.opendev.org/742478 | 19:47 |
| openstackgerrit | wes hayutin proposed openstack/releases master: minor update release for tripleo ussuri https://review.opendev.org/742478 | 20:14 |
| e0ne | ttx, smcginnis: hi. AFAIK, we don't need new xstatic releases because we don't upgrade horizon dependencies. I'll recheck tomorrow | 20:14 |
| smcginnis | Thanks e0ne. I will put up a patch to remove those deliverable files. After you check on things tomorrow, please +1 that if we are good to go ahead with removing them. | 20:32 |
| e0ne | smcginnis: ok, thanks for watching for xstatic-* releases | 20:34 |
| openstackgerrit | wes hayutin proposed openstack/releases master: tripleo victoria-1 release https://review.opendev.org/742492 | 20:36 |
| *** e0ne has quit IRC | 20:41 | |
| *** dave-mccowan has joined #openstack-release | 20:41 | |
| *** jbadiapa has quit IRC | 22:55 | |
| *** tosky has quit IRC | 22:59 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!