| *** ykarel_ is now known as ykarel | 04:09 | |
| ttx | Happy release day! | 06:43 |
|---|---|---|
| ttx | Wearing my "Well, it worked in DevStack" t-shirt to celebrate | 06:43 |
| opendevreview | Rodolfo Alonso proposed openstack/releases master: Release stable/2026.1 ovsdbapp 2.16.1 https://review.opendev.org/c/openstack/releases/+/979460 | 06:59 |
| frickler | CI is looking fine afaict, so start the countdown ;) | 08:32 |
| elodilles | yepp, so far so good | 08:34 |
| elodilles | fingers crossed :] | 08:34 |
| ttx | Patch is ready to W+1 | 09:44 |
| elodilles | yepp | 09:46 |
| elodilles | the plan is to +W at 10:00 UTC, but we can do it sooner if you think | 09:47 |
| fungi | okay, i'm awake(ish) | 09:48 |
| elodilles | good morning fungi o/ | 09:49 |
| elodilles | "All Systems Operational" according to status.pypi.org | 09:49 |
| elodilles | https://status.python.org/ | 09:49 |
| elodilles | to be precise... | 09:49 |
| elodilles | zuul & gerrit seems to be fine, too | 09:50 |
| fungi | i agree things look healthy in zuul | 09:50 |
| fungi | i love the smell of new releases in the morning | 09:51 |
| elodilles | 🚬 :) | 09:51 |
| elodilles | kolla doesn't have too many patches on the gate either o:) | 09:52 |
| elodilles | which is a good opportunity to start the release :D | 09:52 |
| elodilles | okay, let's do it. i'm impatient o:) | 09:55 |
| * elodilles +W'd the release patch | 09:56 | |
| fungi | yay! | 09:56 |
| fungi | zuul estimates we're 2 minutes from merging | 10:05 |
| elodilles | yeah, but it's based on the usual release patches and this has many releases, so validate job will take its time o:) | 10:07 |
| elodilles | 2025.2 Flamingo release patch finished 32 mins in October :) | 10:07 |
| elodilles | and i think we had worse than that before. once it took like 50+ mins | 10:08 |
| fungi | yeah, i think zuul underestimates the runtime for releases-tox-validate because that job doesn't normally have to iterate over as many projects | 10:09 |
| elodilles | yepp | 10:10 |
| fungi | watching the log stream gives a better indication of progress | 10:10 |
| fungi | we're ~halfway through the alphabet | 10:11 |
| elodilles | yepp, we are at letter 'n' so far :] | 10:13 |
| elodilles | though 'o' and 'p' is a bit overrepresented i think | 10:14 |
| elodilles | but things are moving | 10:14 |
| elodilles | the soup is "cooking" (if i can say that to a cold soup o:)) | 10:16 |
| ttx | Sean Michael Kerner said his Gazpacho article will be full of soup puns | 10:17 |
| elodilles | :) | 10:18 |
| fungi | slurp-slurp! | 10:20 |
| fungi | i trust smk's comedy routine | 10:21 |
| fungi | validation has reached the end of the alphabet | 10:22 |
| elodilles | zun | 10:23 |
| elodilles | should be the last? | 10:24 |
| frickler | congratulations :) (1518.81 seconds) | 10:24 |
| fungi | yes | 10:24 |
| elodilles | it's still running by me :-o | 10:24 |
| opendevreview | Merged openstack/releases master: 2026.1 Gazpacho final releases for cycle-with-rc projects https://review.opendev.org/c/openstack/releases/+/982438 | 10:24 |
| elodilles | there it is ^^^ \o/ | 10:25 |
| elodilles | now the post jobs | 10:25 |
| fungi | this time you can even see the node request for tag-releases in action on the requests tab | 10:28 |
| elodilles | :-o | 10:28 |
| fungi | accepted and building in rax-ord | 10:28 |
| fungi | i guess nova is being a little slow there | 10:29 |
| fungi | but the job's finally running now | 10:32 |
| fungi | adjutant-ui has entered the release pipeline | 10:33 |
| fungi | followed by a lot more | 10:33 |
| elodilles | https://zuul.opendev.org/t/openstack/status/pipeline/tag | 10:34 |
| elodilles | deliverables are popping up :-o | 10:34 |
| fungi | https://pypi.org/project/adjutant-ui/#files has our uploaded sdist and wheel | 10:37 |
| fungi | now a few seconds old | 10:37 |
| elodilles | adjutant-ui -- Wed, 01 Apr 2026 10:36:57 GMT -- 12.0.0 | 10:38 |
| elodilles | yepp | 10:38 |
| elodilles | and yepp, release pipeline is also increasing, now 42 items: https://zuul.opendev.org/t/openstack/status/pipeline/release | 10:45 |
| elodilles | #50 and 15 events :-o | 10:50 |
| elodilles | (sorry, that was 'result events' :)) | 10:51 |
| elodilles | 55 | 10:51 |
| elodilles | zun is on the list and now the count is slowly decreasing | 10:54 |
| elodilles | the count is 10 and i think that missing-releases can be started as the relevant jobs have finished | 11:05 |
| elodilles | (the remaining jobs are 'announce-release' and 'propose-update-constraints' jobs) | 11:06 |
| frickler | running | 11:07 |
| fungi | i think we'll have at least the manila rc1 missing from this cycle that's already known | 11:08 |
| fungi | (because pypi wouldn't accept it) | 11:08 |
| elodilles | well, the script checks for the latest releases, so manila final release should be fine | 11:09 |
| elodilles | manila -- Wed, 01 Apr 2026 11:01:04 GMT -- 22.0.0 | 11:09 |
| fungi | ah okay, i thought this was the one that checked for missing files from all releases | 11:09 |
| elodilles | fungi: https://paste.opendev.org/show/bhsyYPItEeBcWdaLlKIE/ | 11:12 |
| fungi | yep cool, thanks | 11:13 |
| ttx | breaking for lunch, brb | 11:14 |
| elodilles | bon apetit o/ | 11:15 |
| elodilles | release pipeline is empty | 11:15 |
| elodilles | tag pipeline still has nova in it :) | 11:15 |
| elodilles | publish-openstack-releasenotes-python3 is running there for almost 30 mins now | 11:16 |
| fungi | nova's reno runs are massive | 11:16 |
| frickler | no missing-releases issues found for me | 11:20 |
| frickler | elodilles: ready for https://review.opendev.org/c/openstack/openstack-manuals/+/981758 then? or wait for nova renos to be done? | 11:22 |
| frickler | iiuc they have a lot of translations | 11:24 |
| fungi | no new messages in https://lists.openstack.org/archives/list/release-job-failures@lists.openstack.org/ so as long as the nova release notes build succeeds we appear to be in the clear | 11:28 |
| frickler | currently building ko_KR, not sure how many langs still come after that | 11:29 |
| frickler | oh, was the final one | 11:29 |
| frickler | now waiting for elodilles to unblock the docs update | 11:32 |
| elodilles | yepp. nova reno job has finished \o/ | 11:35 |
| elodilles | and no missing-releases in my run either | 11:36 |
| elodilles | (well, two false alarm i think: | 11:36 |
| elodilles | did not find python 3 wheel https://tarballs.openstack.org/ansible-role-atos-hsm/ansible_role_atos_hsm-11.0.0-py3-none-any.whl | 11:36 |
| elodilles | did not find python 3 wheel https://tarballs.openstack.org/ansible-role-thales-hsm/ansible_role_thales_hsm-11.0.0-py3-none-any.whl | 11:36 |
| elodilles | but these are "known" issues afair) | 11:36 |
| elodilles | frickler: yes, i think you can +W the openstack-manuals patch, though we have plenty of time fortunately | 11:38 |
| frickler | well those have combined wheels still, so not an issue? they also have no change merged for multiple cycles, which confused me with the "diff-start" anomaly once again (like every cycle I need to dig deep to remember what is going on there) | 11:38 |
| elodilles | yes, unfortunately those are quite abandoned repos :/ | 11:40 |
| elodilles | next cycle we shouldn't even release them if possible | 11:40 |
| fungi | if they're really abandoned, i wonder if they can be retired | 11:43 |
| elodilles | i don't know, but given it haven't got any patches over multiple cycles now... | 11:49 |
| elodilles | okay, the openstack-manuals patch merged >>> https://review.opendev.org/c/openstack/openstack-manuals/+/981758 | 11:51 |
| elodilles | so ~1 hr from now and things will be updated on docs.o.o, right? | 11:52 |
| fungi | likely | 11:53 |
| fungi | based on previous releases | 11:53 |
| fungi | promote-openstack-manuals in the promote pipeline is the job responsible | 11:54 |
| elodilles | and it's running for 21 minutes already | 12:02 |
| elodilles | s/21/14 | 12:03 |
| elodilles | 21 was the remaining minutes | 12:03 |
| elodilles | promote job has finished | 12:29 |
| frickler | "Currently viewing 2026.1 which is the current supported release." yay | 12:31 |
| frickler | ttx: when you're done lunching you could give https://review.opendev.org/c/openstack/releases/+/982898 a second vote? | 12:33 |
| ttx | yes | 12:43 |
| ttx | done | 12:44 |
| frickler | cool, let's hope the current crawler attacks don't hamper this | 12:56 |
| elodilles | :S | 12:59 |
| frickler | there goes hoping, mostly everything failed https://zuul.opendev.org/t/openstack/build/c0cc411c15d04bb4933ca0dcb7917ca7 | 12:59 |
| elodilles | /o\ | 13:00 |
| elodilles | should we wait or should we recheck immediately? O.o | 13:00 |
| ttx | GNU TLS handshake? sounds like a one-off? | 13:00 |
| ttx | I'd recheck, given the tshirt I'm wearing | 13:01 |
| ttx | done | 13:01 |
| fungi | yeah, all our gitea servers got overrun with some sort of crawler activity in the past 30 minutes. i'm not hopeful that recheck is going to fare any better | 13:02 |
| ttx | hah | 13:02 |
| fungi | the backends are all slammed and not returning content | 13:02 |
| ttx | I'm feeling lucky! | 13:02 |
| ttx | headliens: "AI prevented OpenStack from being released on time for first time in 15 years" | 13:03 |
| elodilles | we were so close to a release without any issues... /o\ | 13:04 |
| elodilles | should we prepare a patch to noop the failing jobs for the patch to be able to merge? | 13:05 |
| ttx | I'm not sure that noop patch would be able to land under current conditions | 13:08 |
| ttx | It's one of those times I wish our CI jobs used a gitea server that is not part of the public rotation | 13:09 |
| ttx | only_for_us_gitea.opendev.org | 13:10 |
| opendevreview | Elod Illes proposed openstack/releases master: [CI] Temporarily reduce zuul jobs number https://review.opendev.org/c/openstack/releases/+/983055 | 13:16 |
| elodilles | i mean something like this ^^^ | 13:16 |
| elodilles | what we are missing now is releases.o.o update only. (though it's bad if the pages cannot be reached when the release is announced.... O.o) | 13:19 |
| opendevreview | Elod Illes proposed openstack/releases master: [CI] Temporarily reduce zuul jobs number https://review.opendev.org/c/openstack/releases/+/983055 | 13:21 |
| ttx | Let's wait for fungi's assessment | 13:21 |
| elodilles | +1 | 13:22 |
| fungi | yes, i think frickler is trying to work out the offenders' ip addresses so we can try to temporarily firewall them | 13:23 |
| fungi | i'm looking at porting the anubis implementation we just added to the mailing list server to our gitea backends | 13:23 |
| opendevreview | Elod Illes proposed openstack/releases master: [CI] Temporarily reduce zuul jobs number https://review.opendev.org/c/openstack/releases/+/983055 | 13:46 |
| opendevreview | Elod Illes proposed openstack/releases master: [CI] Temporarily reduce zuul jobs number https://review.opendev.org/c/openstack/releases/+/983055 | 13:51 |
| Clark[m] | What part of the process relies on gitea? Historically, we've tried really hard to ensure zuul jobs don't need to talk to gitea for things but I know Openstack in particular hasn't always been great about following those rules. In particular zuul speaks to Gerrit and manages its own got repo caches and got repo state which is pushed into the tests nodes so you should never need to talk to gitea in a zuul job unless the point of the | 14:01 |
| Clark[m] | zuul job was to check gitea itself | 14:01 |
| frickler | Clark[m]: the tox jobs, not sure about the details currently, e.g. https://zuul.opendev.org/t/openstack/build/c0cc411c15d04bb4933ca0dcb7917ca7 | 14:03 |
| gthiemonge | Hi folks, it was reported to me that the gunicorn release in upper-constraints in gazpacho has a regression that triggers deadlocks (https://github.com/benoitc/gunicorn/discussions/3509), it's fixed in recent versions. I have prepared a workaround for octavia (for master and 2026.1), we will be able to remove the workaround when the new release is included in requirements on master. but is | 14:15 |
| gthiemonge | there a way to update the requirements for 2026.1 (post-release of course)? | 14:15 |
| frickler | gthiemonge: this is more of a question for the requirements team, but yes, under the given circumstances I think we could do it. I'm assuming this is causing sporadic testing failures? | 14:25 |
| zigo | I'm done packaging final versions. Zero change with latest RCs this year. | 14:26 |
| clarkb | frickler: sorry had to step away for a few, but that link is helpful | 14:28 |
| fungi | i'm going to bypass testing to merge 982898 since we have confidence that it was passing check jobs earlier | 14:29 |
| gthiemonge | frickler: yeah I opened a launchpad for octavia https://bugs.launchpad.net/octavia/+bug/2146523 it triggers timeouts in our tests, but the impact might be more limited for the end users, like a delay in the creation of a load balancer | 14:29 |
| clarkb | frickler: and the rest of the release team. It looks like that failing job is fetching git tags then confirming the tags are pushed? Is that something we can consider good enough/done if we run that against gerrit instead of gitea? | 14:29 |
| clarkb | I suspect that we might end up rerunning replication to gitea anyway with the servers there being unhappy so I'm not sure we need to double check the git tag state on gitea? | 14:29 |
| ttx | clarkb: we could | 14:29 |
| gthiemonge | frickler: I forgot that requirements has its own channel, I'll ask them there | 14:30 |
| fungi | the releases were already validated via manual script run after all the tags were done, so should be fine regardless | 14:30 |
| ttx | for the release missing patch we really don't need testing | 14:30 |
| clarkb | got it | 14:31 |
| ttx | fungi: are you still working on a workaround, or should we just merge Elod's test-skipping thing temporarily | 14:31 |
| ttx | since we are 30min away | 14:31 |
| elodilles | i guess fungi can admin-approve the patch to skip testing, maybe in that sense my patch is not needed | 14:32 |
| elodilles | but both could work as a workaround for now | 14:32 |
| ttx | oh yes, force-approving would be easier | 14:33 |
| ttx | rather than disable, merge, enable | 14:33 |
| ttx | I triple-checked it and it is rather safe | 14:34 |
| opendevreview | Merged openstack/releases master: Mark 2026.1 Gazpacho as released https://review.opendev.org/c/openstack/releases/+/982898 | 14:35 |
| opendevreview | Elod Illes proposed openstack/releases master: DNM: release-test with required-repo https://review.opendev.org/c/openstack/releases/+/983074 | 14:36 |
| ttx | alright we are a go | 14:40 |
| elodilles | yepp, patch has merged. though releases.o.o is not updated yet | 14:41 |
| ttx | elodilles: you can send the announcement(s) as soon as that publishes | 14:42 |
| -opendevstatus- NOTICE: The opendev.org site is currently experiencing overwhelming load adversely impacting git operations and repository browsing since 12:20 UTC today, mitigation work is in progress | 14:42 | |
| elodilles | ttx: ACK, prepared the mails, feel free to rephrase if you want to add something: https://etherpad.opendev.org/p/relmgmt-weekly-emails | 14:43 |
| fungi | publish-tox-docs-releases failed in post | 14:47 |
| frickler | publishing seems to have worked, though it still says "2026-04-01 estimated" as initial release date? but at least "maintained" | 14:47 |
| elodilles | frickler: nope, it's not updated, the next state is where it shows "maintained" :/ | 14:48 |
| frickler | ah, wrong column, meh | 14:49 |
| fungi | looking at https://zuul.opendev.org/t/openstack/build/fdfe17a2031f4b6f8f9f1bd085a7254c it's not clear to me what url it was trying to request | 14:49 |
| ttx | lgtm | 14:50 |
| ttx | please send! | 14:51 |
| ttx | (added a "more contributors" remark) | 14:51 |
| fungi | i'm standing by for openstack-announce moderation | 14:51 |
| ttx | still waiting on the docs refresh run right | 14:52 |
| fungi | ttx: no, the publish-tox-docs-releases job failed | 14:53 |
| ttx | ah | 14:54 |
| fungi | "Task Run tox without tests failed..." with a traceback indicating pip hit an ssl connection error but doesn't say what url it tried to hit | 14:54 |
| ttx | hmm, maybe simpler at this stage to consider it "released" even if the website still says "estimated", since we don't really know when that will be fixed | 14:55 |
| ttx | so my vote is on sending the email so that the 10am CT PR does not go out before we announce | 14:56 |
| ttx | elodilles: ^ | 14:57 |
| elodilles | fungi: meanwhile, i've sent the mail to 'announce' ML, feel free to moderate-unblock it when things looks acceptable | 14:57 |
| fungi | announcement accepted in moderation queue now | 14:58 |
| ttx | and you can send to the discuss list as well | 14:59 |
| elodilles | will do as soon as I can see the mail in ML and can add the link in the mail | 14:59 |
| ttx | ++ | 14:59 |
| elodilles | ok, so there is the link: https://lists.openstack.org/archives/list/openstack-announce@lists.openstack.org/thread/7WU5RWBRB4UQQAIHMTCKMTIEDRATHVGD/ | 15:00 |
| ttx | confirmed | 15:00 |
| elodilles | and mail sent to openstack-discuss | 15:02 |
| fungi | as soon as we get the gitea servers responding again, i'm going to reenqueue the failed releases site change on the assumption it was related to the outage | 15:02 |
| elodilles | there it is: https://lists.openstack.org/archives/list/openstack-discuss@lists.openstack.org/thread/Y5DZJV2DZLZ6PQ2HIRFGGJY2Y6ODELTW/ | 15:03 |
| fungi | at least we got the mailing list server responding again, thanks to clarkb's change to slap anubis in front of the service | 15:06 |
| clarkb | fungi: I wish that solutions like that felt like a victory | 15:07 |
| fungi | in a war of attrition, every victory feels like a defeat | 15:08 |
| sean-k-mooney | elodilles: https://releases.openstack.org/index.html has not been updated yet to relfect the fact now in Maintained status instead of development | 15:18 |
| sean-k-mooney | fungi: oh is that related to the failed release site change message above? | 15:19 |
| clarkb | yes that is a known issue | 15:19 |
| sean-k-mooney | ack | 15:19 |
| fungi | as soon as gitea is responding again (what we think but can't confirm caused the releases site publication job to fail because the tox logs don't say what url pip was trying to fetch) i'll ask zuul to rerun it and that should get it updated | 15:20 |
| sean-k-mooney | out of interest is there a reason we use https://releases.openstack.org/gazpacho/ instead of https://releases.openstack.org/2026.1/ | 15:20 |
| sean-k-mooney | we adopted the offical release number in favor of the name in most other places | 15:21 |
| sean-k-mooney | it woudl make finding the release sechdule ectra easier | 15:21 |
| elodilles | sean-k-mooney: release tooling still uses the 'name' instead of the 'id' | 15:26 |
| sean-k-mooney | yep which i personally find harder to work with | 15:28 |
| sean-k-mooney | but also have not had tiem to even consider what it woudl take to change the release repo ectra to use the number instead of the name | 15:29 |
| sean-k-mooney | i assume that part fo why it was not doen no one has had time to think about it properly | 15:29 |
| fungi | 2c28ee8 is running again in release-post again now | 15:29 |
| fungi | publish-tox-docs-releases got past the point where it failed last time | 15:34 |
| fungi | updated releases site content was copied successfully into afs, next vos release will happen in ~4 minutes | 15:35 |
| frickler | https://releases.openstack.org/ does look way more correct now | 15:45 |
| frickler | so ... party time? \o/ | 15:45 |
| elodilles | ~o~ | 15:54 |
| elodilles | fancy :) | 15:54 |
| ttx | Looks like it | 16:03 |
| ttx | congrats! | 16:03 |
| ttx | OpenStack map has been updated to match release contents too | 16:03 |
| sean-k-mooney | indeed it does | 16:12 |
| -opendevstatus- NOTICE: Load on the opendev.org Gitea backends is under control again for now, if any Zuul jobs failed with SSL errors or disconnects reaching the service they can be safely rechecked | 16:13 | |
Generated by irclog2html.py 4.1.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!