| opendevreview | Tim Burke proposed openstack/releases master: Release Swift 2.35.1 https://review.opendev.org/c/openstack/releases/+/958377 | 01:36 |
|---|---|---|
| opendevreview | Merged openstack/releases master: Add dates for 2026.1/Gazpacho elections https://review.opendev.org/c/openstack/releases/+/958225 | 07:10 |
| opendevreview | Elod Illes proposed openstack/releases master: Release final osc-placement for 2025.2 Flamingo https://review.opendev.org/c/openstack/releases/+/958396 | 09:03 |
| opendevreview | Elod Illes proposed openstack/releases master: Release final python-adjutantclient for 2025.2 Flamingo https://review.opendev.org/c/openstack/releases/+/958397 | 09:04 |
| opendevreview | Elod Illes proposed openstack/releases master: Release final python-aodhclient for 2025.2 Flamingo https://review.opendev.org/c/openstack/releases/+/958398 | 09:34 |
| opendevreview | Elod Illes proposed openstack/releases master: Release final python-barbicanclient for 2025.2 Flamingo https://review.opendev.org/c/openstack/releases/+/958399 | 09:39 |
| elodilles | release-team: nova team has 3 stable release patches that they are eagerly waiting (all three contain a sec related fix), when you have time please review them as they need a 2nd core review: https://review.opendev.org/q/project:openstack/releases+is:open+intopic:nova | 09:51 |
| ttx | I'll look into it | 09:54 |
| opendevreview | Elod Illes proposed openstack/releases master: Release final python-brick-cinderclient-ext for 2025.2 Flamingo https://review.opendev.org/c/openstack/releases/+/958400 | 09:54 |
| elodilles | ttx: thanks in advance o/ | 09:54 |
| opendevreview | Elod Illes proposed openstack/releases master: Release final python-cinderclient for 2025.2 Flamingo https://review.opendev.org/c/openstack/releases/+/958401 | 09:55 |
| opendevreview | Elod Illes proposed openstack/releases master: Release final python-cloudkittyclient for 2025.2 Flamingo https://review.opendev.org/c/openstack/releases/+/958402 | 09:56 |
| opendevreview | Elod Illes proposed openstack/releases master: Release final python-cyborgclient for 2025.2 Flamingo https://review.opendev.org/c/openstack/releases/+/958403 | 09:56 |
| opendevreview | Elod Illes proposed openstack/releases master: Release final python-freezerclient for 2025.2 Flamingo https://review.opendev.org/c/openstack/releases/+/958404 | 09:58 |
| opendevreview | Elod Illes proposed openstack/releases master: Release final python-glanceclient for 2025.2 Flamingo https://review.opendev.org/c/openstack/releases/+/958405 | 09:59 |
| opendevreview | Elod Illes proposed openstack/releases master: Release final python-ironic-inspector-client for 2025.2 Flamingo https://review.opendev.org/c/openstack/releases/+/958406 | 10:00 |
| opendevreview | Elod Illes proposed openstack/releases master: Release final python-ironicclient for 2025.2 Flamingo https://review.opendev.org/c/openstack/releases/+/958407 | 10:01 |
| opendevreview | Elod Illes proposed openstack/releases master: Release final python-keystoneclient for 2025.2 Flamingo https://review.opendev.org/c/openstack/releases/+/958409 | 10:02 |
| opendevreview | Elod Illes proposed openstack/releases master: Release final python-magnumclient for 2025.2 Flamingo https://review.opendev.org/c/openstack/releases/+/958410 | 10:02 |
| opendevreview | Elod Illes proposed openstack/releases master: Release final python-manilaclient for 2025.2 Flamingo https://review.opendev.org/c/openstack/releases/+/958411 | 10:02 |
| opendevreview | Elod Illes proposed openstack/releases master: Release final python-masakariclient for 2025.2 Flamingo https://review.opendev.org/c/openstack/releases/+/958412 | 10:03 |
| opendevreview | Elod Illes proposed openstack/releases master: Release final python-mistralclient for 2025.2 Flamingo https://review.opendev.org/c/openstack/releases/+/958413 | 10:05 |
| opendevreview | Elod Illes proposed openstack/releases master: Release final python-novaclient for 2025.2 Flamingo https://review.opendev.org/c/openstack/releases/+/958414 | 10:06 |
| opendevreview | Merged openstack/releases master: nova: Release 2025.1 Epoxy 31.1.0 https://review.opendev.org/c/openstack/releases/+/957951 | 10:06 |
| opendevreview | Merged openstack/releases master: nova: Release 2024.2 Dalmatian 30.1.0 https://review.opendev.org/c/openstack/releases/+/947847 | 10:07 |
| opendevreview | Merged openstack/releases master: nova: Release 2024.1 Caracal 29.3.0 https://review.opendev.org/c/openstack/releases/+/958009 | 10:07 |
| opendevreview | Elod Illes proposed openstack/releases master: Release final python-observabilityclient for 2025.2 Flamingo https://review.opendev.org/c/openstack/releases/+/958415 | 10:07 |
| opendevreview | Elod Illes proposed openstack/releases master: Release final python-octaviaclient for 2025.2 Flamingo https://review.opendev.org/c/openstack/releases/+/958416 | 10:09 |
| opendevreview | Elod Illes proposed openstack/releases master: Release final python-tackerclient for 2025.2 Flamingo https://review.opendev.org/c/openstack/releases/+/958417 | 10:10 |
| opendevreview | Elod Illes proposed openstack/releases master: Release final python-troveclient for 2025.2 Flamingo https://review.opendev.org/c/openstack/releases/+/958418 | 10:10 |
| opendevreview | Elod Illes proposed openstack/releases master: Release final python-venusclient for 2025.2 Flamingo https://review.opendev.org/c/openstack/releases/+/958419 | 10:11 |
| opendevreview | Elod Illes proposed openstack/releases master: Release final python-vitrageclient for 2025.2 Flamingo https://review.opendev.org/c/openstack/releases/+/958420 | 10:12 |
| opendevreview | Elod Illes proposed openstack/releases master: Release final python-watcherclient for 2025.2 Flamingo https://review.opendev.org/c/openstack/releases/+/958421 | 10:12 |
| opendevreview | Elod Illes proposed openstack/releases master: Release final python-zaqarclient for 2025.2 Flamingo https://review.opendev.org/c/openstack/releases/+/958422 | 10:14 |
| opendevreview | Elod Illes proposed openstack/releases master: Release final python-zunclient for 2025.2 Flamingo https://review.opendev.org/c/openstack/releases/+/958423 | 10:15 |
| opendevreview | Merged openstack/releases master: Release final oslo.i18n for 2025.2 Flamingo https://review.opendev.org/c/openstack/releases/+/957718 | 12:37 |
| opendevreview | Merged openstack/releases master: Release final os-brick for 2025.2 Flamingo https://review.opendev.org/c/openstack/releases/+/957706 | 12:38 |
| opendevreview | Merged openstack/releases master: Release final stevedore for 2025.2 Flamingo https://review.opendev.org/c/openstack/releases/+/957740 | 12:38 |
| opendevreview | Merged openstack/releases master: Release final oslo.privsep for 2025.2 Flamingo https://review.opendev.org/c/openstack/releases/+/957723 | 12:41 |
| opendevreview | Merged openstack/releases master: Release final kuryr for 2025.2 Flamingo https://review.opendev.org/c/openstack/releases/+/957703 | 12:41 |
| opendevreview | Merged openstack/releases master: Release final glance_store for 2025.2 Flamingo https://review.opendev.org/c/openstack/releases/+/957747 | 12:41 |
| opendevreview | Merged openstack/releases master: Release final oslo.serialization for 2025.2 Flamingo https://review.opendev.org/c/openstack/releases/+/957730 | 12:41 |
| opendevreview | Merged openstack/releases master: Release final oslo.rootwrap for 2025.2 Flamingo https://review.opendev.org/c/openstack/releases/+/957729 | 12:42 |
| opendevreview | Merged openstack/releases master: Release final oslo.metrics for 2025.2 Flamingo https://review.opendev.org/c/openstack/releases/+/957721 | 12:42 |
| opendevreview | Merged openstack/releases master: Release final oslo.versionedobjects for 2025.2 Flamingo https://review.opendev.org/c/openstack/releases/+/957737 | 12:42 |
| opendevreview | Merged openstack/releases master: Release final oslo.context for 2025.2 Flamingo https://review.opendev.org/c/openstack/releases/+/957716 | 12:42 |
| opendevreview | Merged openstack/releases master: Release final oslo.reports for 2025.2 Flamingo https://review.opendev.org/c/openstack/releases/+/957728 | 12:43 |
| opendevreview | Merged openstack/releases master: Release final oslo.policy for 2025.2 Flamingo https://review.opendev.org/c/openstack/releases/+/957722 | 12:44 |
| opendevreview | Merged openstack/releases master: Release final oslo.utils for 2025.2 Flamingo https://review.opendev.org/c/openstack/releases/+/957736 | 12:44 |
| opendevreview | Merged openstack/releases master: Release final oslo.upgradecheck for 2025.2 Flamingo https://review.opendev.org/c/openstack/releases/+/957734 | 12:44 |
| opendevreview | Merged openstack/releases master: Release final taskflow for 2025.2 Flamingo https://review.opendev.org/c/openstack/releases/+/957746 | 12:44 |
| opendevreview | Merged openstack/releases master: Release final oslo.concurrency for 2025.2 Flamingo https://review.opendev.org/c/openstack/releases/+/957715 | 12:45 |
| opendevreview | Merged openstack/releases master: Release final oslo.log for 2025.2 Flamingo https://review.opendev.org/c/openstack/releases/+/957720 | 12:45 |
| opendevreview | Merged openstack/releases master: Release final oslo.cache for 2025.2 Flamingo https://review.opendev.org/c/openstack/releases/+/957714 | 12:48 |
| opendevreview | Merged openstack/releases master: Release final neutron-lib for 2025.2 Flamingo https://review.opendev.org/c/openstack/releases/+/957704 | 12:49 |
| opendevreview | Merged openstack/releases master: Release final ovsdbapp for 2025.2 Flamingo https://review.opendev.org/c/openstack/releases/+/957738 | 12:51 |
| opendevreview | Merged openstack/releases master: Release final os-ken for 2025.2 Flamingo https://review.opendev.org/c/openstack/releases/+/957710 | 12:53 |
| opendevreview | Merged openstack/releases master: Release final os-vif for 2025.2 Flamingo https://review.opendev.org/c/openstack/releases/+/957711 | 12:53 |
| opendevreview | Elod Illes proposed openstack/releases master: Release final oslo.service for 2025.2 Flamingo https://review.opendev.org/c/openstack/releases/+/957733 | 14:16 |
| opendevreview | Merged openstack/releases master: Release Mistral 20.1.0 https://review.opendev.org/c/openstack/releases/+/952201 | 14:46 |
| opendevreview | Merged openstack/releases master: Release Mistral 19.1.0 https://review.opendev.org/c/openstack/releases/+/952202 | 14:51 |
| opendevreview | Merged openstack/releases master: Release Mistral 18.1.0 https://review.opendev.org/c/openstack/releases/+/952203 | 14:51 |
| ttx | There were a bunch of errors but I have no time to investigate them... | 15:18 |
| elodilles | oh, ACK, thanks, i'll take a look now | 15:19 |
| elodilles | it seems to me that there was a some kind of infra outage around 14:47-14:57 UTC | 15:25 |
| elodilles | clarkb: ^^^ were there any infra activity around this time? | 15:28 |
| elodilles | there's no logs for the broken jobs :/ https://lists.openstack.org/archives/list/release-job-failures@lists.openstack.org/ | 15:30 |
| elodilles | here are some examples: https://zuul.opendev.org/t/openstack/builds?job_name=tag-releases&skip=0 | 15:30 |
| elodilles | and https://zuul.opendev.org/t/openstack/builds?job_name=openstack-upload-github-mirror&skip=0 | 15:30 |
| elodilles | https://zuul.opendev.org/t/openstack/builds?job_name=publish-openstack-releasenotes-python3&skip=0 | 15:31 |
| elodilles | can we somehow re-enqueue these jobs? | 15:33 |
| clarkb | elodilles: I'm catching up on like 3 things since the weekend. Can you link to an example failure? | 16:12 |
| clarkb | if I can focus on one thing to look into that will be helpful | 16:12 |
| clarkb | then we can decide if we can reeqneue | 16:12 |
| clarkb | https://zuul.opendev.org/t/openstack/build/b5f6d9886def44afb387899395c6bcf4 is this an example? | 16:14 |
| clarkb | the problem with those high level links is it returns a big list of jobs some of which succeeded and some failed. its hard to know if any of those failures were expecte or debugged or maybe the problems are even in successful jobs. It helps to have something concrete | 16:15 |
| clarkb | elodilles: I believe that https://zuul.opendev.org/t/openstack/build/b5f6d9886def44afb387899395c6bcf4 this build successfully updated the github mirror then failed to upload its own job logs to swift | 16:37 |
| clarkb | of the three job lists that you linked I suspect tag-releases is the only one where normal followup processing might not resynchronize us and catch us up (github mirroring can be rerun by the next thing that lands, as can release notes publishing) | 16:39 |
| clarkb | so I'm going to focus on those next and determine if they were successful other than reporting their build logs | 16:39 |
| clarkb | https://opendev.org/openstack/mistral/src/tag/19.1.0/ and https://opendev.org/openstack/mistral/src/tag/20.1.0/ both appear to exist and would've been created by https://zuul.opendev.org/t/openstack/build/e36657b866ef48308e5211d878cece0a and https://zuul.opendev.org/t/openstack/build/57055d1d30c74b9faf60d9c9a731d907 | 16:40 |
| clarkb | e3665 also failed to upload its logs but the run release scripts task (and its parent playbook) appear to ahve succeeded. It ran on ze03 | 16:42 |
| *** melwitt_ is now known as jgwentworth | 16:43 | |
| *** jgwentworth is now known as melwitt | 16:44 | |
| clarkb | same thing with 57055d except it ran on ze12 | 16:44 |
| clarkb | all three jobs I looked at attempted to upload their logs to swift in ovh gra. I don't see a see of orange on the current zuul status page which makes me think that this was a short blip and isn't a persisntent issue. However, if it comes back I can disable that region for log uploads | 16:45 |
| clarkb | for releases as far as I can tell the job payloads ran and simply failed to upload their own logs. The mistral tags exist etc. I would suggest that you check all of your artifacts and side effects are as expected but I do not think we should reenqueue any of these jobs as they did run | 16:45 |
| clarkb | if the logs themselves contain important info that you need I can try to fish that out of the executor debug logs | 16:50 |
| opendevreview | sean mooney proposed openstack/releases master: Release watcher stable branches for bug 2112187 https://review.opendev.org/c/openstack/releases/+/958468 | 19:27 |
| elodilles | clarkb: ACK, thanks for the investigation, i will double check all of the artifacts tomorrow. the problem is that some job failures can cause that the post jobs are not starting at all, so i'm sure there will be some missing things... but let's hope i'm wrong | 20:02 |
| clarkb | elodilles: that is helpful. Looks like publish-tox-docs-releases did not run when the tag-releases jobs failed. However, a later buildset did run and ran that job so its probably caught up | 20:04 |
| clarkb | https://zuul.opendev.org/t/openstack/build/18ad29a9653148faaf3524575bcb1459 this one for os-vif doesn't seem to have run subsequent jobs so it may still be stale | 20:06 |
| clarkb | but that depends on which jobs would be run | 20:06 |
| clarkb | https://pypi.org/project/os-vif/#files it did get published to pypi | 20:07 |
| clarkb | don't think any of this changes what I reported earlier. It seems like most thingsprobably functioned well enough. If you find something that didn't we can dig in closer and try to correct it | 20:07 |
| elodilles | +1, i'll let you know tomorrow and we will see whether we have to correct anything | 20:11 |
| clarkb | sounds good | 20:12 |
| opendevreview | Jay Faulkner proposed openstack/releases master: [ironic] Cycle highlights for Flamingo/2025.2 https://review.opendev.org/c/openstack/releases/+/958489 | 20:51 |
| *** haleyb is now known as haleyb|out | 20:58 | |
Generated by irclog2html.py 4.0.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!