| opendevreview | Merged openstack/releases master: Add release note links for 2025.2 Flamingo (#4) https://review.opendev.org/c/openstack/releases/+/962582 | 06:29 |
|---|---|---|
| opendevreview | Elod Illes proposed openstack/releases master: Add release note links for 2025.2 Flamingo (#5) https://review.opendev.org/c/openstack/releases/+/962640 | 07:03 |
| ttx | Good Flamingo day everyone | 07:39 |
| elodilles | good morning \o/ | 07:53 |
| elodilles | ~2 hrs and we can start the release machinery :) | 07:54 |
| opendevreview | Merged openstack/releases master: Add release note links for 2025.2 Flamingo (#5) https://review.opendev.org/c/openstack/releases/+/962640 | 07:57 |
| elodilles | let's see if we are ready to push the button! | 09:54 |
| elodilles | python infrastructure seems fine | 09:56 |
| elodilles | gerrit, zuul are reachable, they look fine, too | 09:57 |
| fungi | i'm around | 09:57 |
| elodilles | fungi: o/ | 09:58 |
| elodilles | anyone fancy to push the +W button? | 09:58 |
| elodilles | https://review.opendev.org/c/openstack/releases/+/962461 | 09:58 |
| elodilles | ^^^ | 09:58 |
| elodilles | sry, i was impatient. I've +W'd it o:) | 10:01 |
| ttx | I PTL+1'ed it so you deserved the +W | 10:06 |
| elodilles | :] | 10:08 |
| elodilles | now 30-50 mins until tox-validate job finishes, as far as i remember | 10:08 |
| elodilles | zuul's check and gate piplines have moderate load, by looking at the current items count | 10:10 |
| elodilles | though there are a couple of kolla patches among them. as far as i remember those are more resource-needy | 10:10 |
| fungi | remember that zuul does make an attempt at fair queuing to give priority to projects/queues with fewer recent items | 10:12 |
| elodilles | ah, good then | 10:13 |
| fungi | though the buildset seems to have gotten nodes assigned right away so not a problem regardless | 10:14 |
| elodilles | +1 | 10:14 |
| fungi | https://grafana.opendev.org/d/21a6e53ea4/zuul-status?orgId=1&from=now-6h&to=now&timezone=utc indicates no backlog of node requests and looks like we're only using a fraction of the available quota at the moment anyway | 10:16 |
| elodilles | nice ~o~ | 10:22 |
| frickler | validate: OK (1555.32=setup[5.12]+cmd[7.16,51.68,1491.37] seconds) | 10:32 |
| opendevreview | Merged openstack/releases master: 2025.2 Flamingo final releases for cycle-with-rc projects https://review.opendev.org/c/openstack/releases/+/962461 | 10:32 |
| elodilles | that was quick ^^^ | 10:34 |
| frickler | and tag-releases has started now | 10:34 |
| elodilles | \o/ | 10:34 |
| ttx | lunching | 10:42 |
| fungi | https://pypi.org/project/adjutant-ui/ | 10:45 |
| fungi | releases are starting to appear on pypi | 10:46 |
| tkajinam_ | \o/ | 10:54 |
| *** tkajinam_ is now known as tkajinam | 10:54 | |
| elodilles | yepp, release pipeline queue is shrinking | 10:54 |
| elodilles | 14 | 11:04 |
| elodilles | 6 | 11:06 |
| elodilles | 4 | 11:07 |
| fungi | looking at the release-job-failures archive, nothing yet from today but did anyone look into the glance-tempest-plugin 0.14.0 upload failure last week? | 11:09 |
| fungi | doesn't appear it ever got retried | 11:10 |
| fungi | https://zuul.opendev.org/t/openstack/builds?project=openstack%2Fglance-tempest-plugin&skip=0 | 11:10 |
| fungi | oh, actually it did everything except publish the logs | 11:11 |
| fungi | so probably fine to ignore, it just resulted in the release announcement and constraints proposal jobs getting skipped | 11:12 |
| elodilles | fungi: we discussed it https://meetings.opendev.org/irclogs/%23openstack-release/%23openstack-release.2025-09-23.log.html#openstack-release.2025-09-23.log.html#t2025-09-23T15:19:55 | 11:12 |
| fungi | perfect, i was probably spacing out during that meeting. sorry for the confusion! | 11:13 |
| fungi | release pipeline jobs are done, only the nova release notes build remains in the tag pipeline now | 11:13 |
| elodilles | it wasn't a meeting, this is just IRC log from the channel, just the URL 'meetings.o.o' is a bit misleading o:) | 11:14 |
| fungi | er, yeah. i need to finish this cup o' coffee, clearly ;) | 11:14 |
| elodilles | :] | 11:15 |
| elodilles | enjoy your morning coffee ;) | 11:15 |
| elodilles | ttx: missing-releases task can be started if you are around | 11:15 |
| fungi | right, the only thing that's still not finished is the nova release notes update | 11:16 |
| elodilles | i've also started the script (twice already) and based on the 1st runs failures which are already there, i guess there shouldn't be any missing artifact | 11:17 |
| fungi | wow, nova's release notes really do take a while to build, don't they? 40 minutes and counting | 11:19 |
| elodilles | yes. a bit surprising. | 11:20 |
| elodilles | i checked if it got stuck somewhere, but it isn't... it's just... slow... O.o | 11:20 |
| ttx | checking missing-releases... | 11:22 |
| elodilles | ttx: note that if you encounter some failures, then it might be because this is not yet landed: https://review.opendev.org/c/openstack/releases/+/958536 | 11:23 |
| fungi | and that last remaining tag pipeline item has finished now as well, still nothing new in the release-job-failures archive so we should be in the clear as far as the critical pieces are concerned *phew* | 11:25 |
| elodilles | \o/ | 11:27 |
| elodilles | cool | 11:27 |
| ttx | 21 errors on oslo.* packages, checking manually that the oslo_* wheel version is there | 11:35 |
| fungi | which i guess 958536 would make unnecessary | 11:36 |
| fungi | i've just un-wip'd https://review.opendev.org/957466 too, it can merge whenever as long as it happens no later than monday next week | 11:38 |
| elodilles | yepp, the missing-releases command succeeded to me in the meantime, without a single failure | 11:38 |
| ttx | Alright, they are all here | 11:40 |
| ttx | I'll be back in an hour | 11:41 |
| elodilles | ttx: \o/ ACK, thanks | 11:41 |
| elodilles | next task is to merge the doc page update patch @ 13:00 UTC >>> https://review.opendev.org/c/openstack/openstack-manuals/+/961967 | 11:42 |
| elodilles | but maybe we don't want to wait that much | 11:43 |
| elodilles | nevertheless, this needs people with +2+W rights on openstack-manuals o:) | 11:43 |
| elodilles | if i remember correctly ttx and frickler have these rights | 11:44 |
| fungi | i wouldn't wait too long, having the docs site update early doesn't hurt but also we've run into lengthy delays running that through gate/promote in the past | 11:44 |
| elodilles | yepp | 11:45 |
| elodilles | we have the note: "The doc pages need approximately 1 hour to refresh, so needs to be done well in time!" | 11:45 |
| elodilles | hence the 13:00 UTC target time, but as you say, it doesn't hurt to merge it earlier | 11:46 |
| fungi | yeah, the gate jobs take a while, but the promote jobs take longer than you would expect, and then the vos release in afs can take some time too if there's a lot of content to sync | 11:46 |
| fungi | i don't have approval permissions on openstack-manuals, fwiw | 11:57 |
| elodilles | neither have i | 11:58 |
| elodilles | only TC members and ttx have | 11:58 |
| fungi | looks like frickler just got it | 11:58 |
| fungi | in the gate now | 11:58 |
| frickler | yup | 11:58 |
| elodilles | awesome, thanks \o/ | 11:59 |
| elodilles | btw, maybe this can be merged too: https://review.opendev.org/c/openstack/openstack-manuals/+/962254 | 12:03 |
| fungi | frickler: ^ | 12:04 |
| fungi | also an early warning, soon after the release (possibly even later today) there will be a set of vitrage stable point releases requested for https://storyboard.openstack.org/#!/story/2011539 | 12:07 |
| fungi | zuul's estimating half an hour for the promote job to publish the docs site update | 12:08 |
| elodilles | fungi: ACK, frickler noonedeadpunk and I were discussing it yesterday evening (CEST timezone) | 12:09 |
| ttx | I'll approve https://review.opendev.org/c/openstack/openstack-manuals/+/961967 in 4 minutes | 12:26 |
| fungi | ttx: it's already merged, we're waiting for the content to upload into afs | 12:26 |
| ttx | perfect! | 12:27 |
| fungi | which could still be a while, zuul estimates 9 more minutes, but that may not be accurate | 12:27 |
| fungi | and then we have to wait for the periodic vos release to sync it to the afs read-only replicas that the site serves | 12:28 |
| noonedeadpunk | fwiw, I've just confirmed that proposed patch does cover the reported attack vector | 12:35 |
| noonedeadpunk | I would need some help from stable-maint team to land patches to 2023.1 | 12:36 |
| elodilles | noonedeadpunk: will review it in a minute | 12:38 |
| elodilles | noonedeadpunk: is there any reason behind you didn't propose the fix to stable/2024.2 and stable/2024.1? | 12:39 |
| fungi | promote just finished | 12:44 |
| fungi | vos release willl start in about a minute | 12:44 |
| elodilles | ++ | 12:44 |
| fungi | running now | 12:45 |
| fungi | and done, https://docs.openstack.org/2025.2/index.html no longer says it's in development at the top | 12:45 |
| elodilles | ~o~ | 12:46 |
| elodilles | the release notes link still bring me to 2025.1 Epoxy from 2025.2 page :/ | 12:47 |
| fungi | yeah, looks like frickler approved 962254 but it fell out of the gate, investigating | 12:48 |
| elodilles | oh, the other patch got -2 from Zuul :-o | 12:48 |
| elodilles | i guess the job ran during the time when the page was refreshing for the previous patch | 12:49 |
| elodilles | i guess we can recheck it now that docs.o.o is refreshed, right? | 12:50 |
| fungi | worth a try, yeah | 12:50 |
| elodilles | ok, let's see | 12:51 |
| fungi | it failed running `tox -e publishdocs` with a bunch of http/403 errors, hard to tell why exactly but maybe i can find it in the server logs | 12:51 |
| fungi | but yeah, i have a feeling that there was an interim vos release in the middle of the lengthy content upload, and so those files didn't exist or had incorrect permissions at the time the job was checking them | 12:53 |
| fungi | my guess is permissions or else it would have been a 404 | 12:54 |
| elodilles | yepp | 12:54 |
| elodilles | and the links work now perfectly | 12:55 |
| elodilles | so, fingers crossed | 12:57 |
| noonedeadpunk | elodilles: I am lasy to rebase | 12:59 |
| noonedeadpunk | so waiting for unit tests to merge first | 12:59 |
| noonedeadpunk | *lazy | 12:59 |
| noonedeadpunk | this are coming today as well | 13:00 |
| noonedeadpunk | *these | 13:00 |
| elodilles | ACK, will wait for them, then i'll review um/2023.1 patches | 13:01 |
| fungi | server says this: | 13:01 |
| fungi | [Wed Oct 01 12:30:13.153405 2025] [core:error] [pid 2271305:tid 140609234921024] (110)Connection timed out: [client 2607:5300:201:2000::68e:38280] AH00036: access to /keystone/pike/index.html failed (filesystem path '/afs/openstack.org/docs') | 13:01 |
| fungi | not super helpful, but i think it doesn't contradict our theory at least | 13:01 |
| elodilles | hmmm. again. but now not pike, rather for a couple of ocata links O.o | 13:13 |
| fungi | the list of errors looks far shorter this time. maybe give it one more try? | 13:14 |
| elodilles | done. let's see... | 13:15 |
| fungi | [Wed Oct 01 13:10:13.530593 2025] [core:error] [pid 2272634:tid 140610688648768] (110)Connection timed out: [client 2001:4802:7805:101:be76:4eff:fe20:2c40:45454] AH00036: access to /murano/ocata/index.html failed (filesystem path '/afs/openstack.org/docs') | 13:15 |
| fungi | so same basic error still | 13:15 |
| fungi | i suspect what's actually happening is, with all the files that got touched by the docs site update, afs client on static02 (where the docs vhost is served) is being slow pulling some of the files into its local cache and apache is giving up before the file is returned | 13:17 |
| Clark[m] | You might be able to improve that by listing the files ahead of time on static? | 13:30 |
| fungi | yeah, in the future that's worth considering, though this was an anomalous change in the first place | 13:33 |
| fungi | not a normal step in the release process | 13:34 |
| elodilles | exactly | 13:34 |
| elodilles | we should have merged this patch yesterday or the day before that :/ | 13:34 |
| fungi | it survived check this time so should be fine now | 13:49 |
| fungi | merged one minute ago actually | 13:49 |
| elodilles | ah, good | 13:50 |
| elodilles | let's move on then | 13:50 |
| fungi | promote job is already uploading the fixed files into afs | 13:50 |
| elodilles | i'm about to +W this then: https://review.opendev.org/c/openstack/releases/+/962475 | 13:51 |
| elodilles | to set 2025.2 Flamingo as released on releases.o.o | 13:51 |
| ttx | sounds good | 13:51 |
| fungi | we're a little over an hour from the end of the press release embargo, so should be good timing | 13:51 |
| elodilles | or rather let ttx to do that if you are around o:) | 13:52 |
| ttx | ok I can do that | 13:52 |
| elodilles | \o/ | 13:52 |
| ttx | done | 13:53 |
| elodilles | after this, comes next: Remind Foundation staff to switch openstack.org 2025.2 Flamingo messaging on (ttx) | 13:53 |
| elodilles | and let me prepare the mail contents for openstack-announce and openstack-discuss (won't send it until you signal that they are ready to send) | 13:54 |
| ttx | yes, so the page is ready, but the redirect is still in progress | 13:57 |
| fungi | we just finished reviewing the content | 13:57 |
| opendevreview | Merged openstack/releases master: Mark 2025.2 Flamingo as released https://review.opendev.org/c/openstack/releases/+/962475 | 14:05 |
| elodilles | OK, this one merged as well ^^^ | 14:07 |
| elodilles | here are the mails to send: https://etherpad.opendev.org/p/relmgmt-weekly-emails | 14:07 |
| elodilles | feel free to rephrase or add anything extra info if you want | 14:07 |
| elodilles | and let me know when i can send the 1st one to the 'announce' list, and when that arrives, i'll update the 2nd mail with the thread link and send it to 'discuss' list | 14:09 |
| fungi | i'm also standing by to approve the openstack-announce message when it lands in the moderation queue | 14:10 |
| frickler | mails lgtm | 14:10 |
| elodilles | thx both of you o/ | 14:13 |
| ttx | emails look good! | 14:16 |
| elodilles | thx o/ | 14:16 |
| elodilles | okay, https://releases.openstack.org/ is also refreshed, Flamingo is showed as 'maintained' and Gazpacho as in development | 14:22 |
| elodilles | (flamingo release notes link is not yet refreshed on docs.o.o) | 14:23 |
| ttx | OK let's hold a bit before sending the emails, as the openstack.org website is not switched to Flamingo yet | 14:24 |
| fungi | yeah, the promote job is still running | 14:24 |
| fungi | it's been on "TASK[upload-afs-roots : Synchronize files to AFS]" since 13:49:32 | 14:24 |
| fungi | okay, finished, next afs vos release in 2 minutes | 14:28 |
| elodilles | and the link is correct now \o/ | 14:30 |
| fungi | and done, | 14:30 |
| fungi | yep, confirmed refreshing gives me the right link now | 14:31 |
| fungi | wait, no, i'm still getting the old link | 14:31 |
| fungi | oh, never mind, i was on the wrong page | 14:32 |
| elodilles | for me the refresh helped | 14:32 |
| elodilles | :) | 14:32 |
| fungi | yeah, for some reason when i go to docs.openstack.org my browser is holding onto the old redirect and pulling https://docs.openstack.org/2025.1/ | 14:32 |
| fungi | if i use wget i definitely see the correct redirect now | 14:33 |
| fungi | oh! i see the problem | 14:33 |
| fungi | HTTP request sent, awaiting response... 301 Moved Permanently | 14:33 |
| fungi | we should be using 302 for that, not 301 | 14:34 |
| fungi | i wonder where we need to fix it, .htaccess in openstack-manuals? | 14:34 |
| fungi | yeah, found it, working on a fix | 14:37 |
| fungi | looks like this has been broken for very many years, and it's a bit of a mess of nested macros relying on defaults | 14:38 |
| fungi | i wonder why nobody has spotted this problem before | 14:38 |
| fungi | or maybe they have and gave up after seeing how much work it would be to fix | 14:38 |
| fungi | https://review.opendev.org/c/openstack/openstack-manuals/+/962684 Use HTTP/302 temporary redirects for versions [NEW] | 14:47 |
| ttx | OK I think we can send the release communications now... The openstack.org website still needs one fix but that should not hold us back | 14:47 |
| elodilles | ACK, sending in a minute | 14:47 |
| ttx | (the openstack/org main page shows Flamingo, but the openstack.org/software page still links to Epoxy when you click "latest release" on it) | 14:47 |
| ttx | and that should be fixed in a few minutes | 14:48 |
| elodilles | i see | 14:49 |
| elodilles | fungi: the mail to announce ML is sent, you should see it hopefully | 14:50 |
| fungi | approved | 14:50 |
| elodilles | \o/ | 14:51 |
| ttx | OK, fired up social media | 14:52 |
| elodilles | okay, i have the link to the thread, updating the 2nd mail and sending it in a minute | 14:53 |
| elodilles | https://lists.openstack.org/archives/list/openstack-announce@lists.openstack.org/thread/7ANVEGX7NMVJ7A6ROGCPGXGC7NWQ4UBT/ | 14:53 |
| clarkb | I have recieved ^ in my inbox | 14:54 |
| elodilles | 2nd email sent as well | 14:55 |
| clarkb | I have received that on in openstack-discuss as well | 14:55 |
| elodilles | https://lists.openstack.org/archives/list/openstack-discuss@lists.openstack.org/thread/JICRNVHQPIMYOBWVM5I67N3BE7TZXAHE/ | 14:56 |
| clarkb | for this release do we say 'F' ya! | 14:56 |
| elodilles | clarkb: :D | 14:56 |
| elodilles | awesome!!! thanks everyone \o/ | 14:57 |
| clarkb | do we need to put the semaphore back in place? | 14:59 |
| clarkb | I guess ya'll have a big checklist and i should just wait for pings on things I can be helpful with at this point. But I'm glad it appears to have gone smoothly | 15:00 |
| elodilles | relmgt checklist all done, but i'm not sure that every relnotes thing have refreshed as it should, so i don't know if we can set back the semaphoe or we still should wait some time | 15:02 |
| fungi | clarkb: yeah https://review.opendev.org/962477 | 15:03 |
| fungi | the release notes jobs all finished hours ago | 15:03 |
| clarkb | fungi: re elodilles note about relnotes do we want to hold off on ^ or shoudl I approve it? | 15:03 |
| elodilles | nice | 15:04 |
| clarkb | ok ya I just checked the zuul queues and they are empty | 15:04 |
| clarkb | I'll approve the change now. | 15:04 |
| fungi | the last release notes job was the one in the tag pipeline for nova that took something like 50 minutes to complete | 15:04 |
| fungi | but it's long done | 15:04 |
| elodilles | oh, yeah, the nova reno job that took 40+ mins, sorry, i have a very short memory it seems /o\ | 15:05 |
| fungi | also a reminder that it would be good to go ahead with https://review.opendev.org/957466 soon, it's no longer wip and needs to merge before monday when i'll approve the key rotation that depends-on it | 15:06 |
| fungi | i've already set myself a reminder for the key rotation change | 15:06 |
| elodilles | fungi: +2+W'd it | 15:08 |
| fungi | thanks! | 15:08 |
| elodilles | np | 15:09 |
| opendevreview | Merged openstack/releases master: Publish the 2026.1/Gazpacho key for future use https://review.opendev.org/c/openstack/releases/+/957466 | 15:12 |
Generated by irclog2html.py 4.0.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!