| *** dtantsur_ is now known as dtantsur | 01:54 | |
| tkajinam | I checked the console of a few jobs running in https://zuul.opendev.org/t/openstack/status but all shows "Build ID .... not found". Can it be an issue with zuul ? | 08:38 |
|---|---|---|
| tkajinam | I've updated https://review.opendev.org/c/openstack/heat/+/930826 a few times but there still remains a job for patch set 1. | 08:39 |
| tkajinam | infra-root: ^^^ fyi | 09:20 |
| tkajinam | ok it does not pick up new patches it seems | 09:48 |
| tkajinam | https://zuul.opendev.org/components gives me an impression that something is wrong with zuul01 | 15:44 |
| corvus | i'm looking into that now | 16:51 |
| fungi | thanks! | 16:51 |
| fungi | i started digging in the debug logs on zuul01 but the only exceptions raised around restart seem to be benign (related to looking up default branches in projects during indexing) | 16:52 |
| kevko | Hi, do anybody know why new patchset didn't trigger zuul check pipeline ? Is something broken ? | 16:55 |
| kevko | fungi: Maybe you know something ? ^^ | 16:56 |
| fungi | kevko: that's what we were discussing immediately above | 16:56 |
| kevko | Yeah, I am too fast :/ ..sorry ..just reading | 16:57 |
| kevko | Moreover connected from phone | 16:57 |
| kevko | Apologize | 16:57 |
| fungi | no worries | 16:57 |
| kevko | Btw, it's whole day ..I firstly noticed at the morning | 16:58 |
| fungi | yes, it's been stalled out since around 10:25 utc yesterday, when our automated upgrade continuous deployment ran | 16:58 |
| fungi | or rather, tried to run, but hasn't completed (technically still in-progress, one scheduler is still in an initializing state so the second scheduler hasn't upgraded yet and is still on a prior version) | 16:59 |
| corvus | i have found and temporarily patched the problem; the openstack tenant should resume processing momentarily | 17:17 |
| fungi | awesome, thanks again! | 17:17 |
| corvus | i need more information to actually fix the bug, so i plan on reconfiguring the tenant once more (that may cause another delay, but only while i'm investigating) | 17:18 |
| fungi | makes sense | 17:18 |
| corvus | i'm downing zuul-web on zuul01 since that will have the same problem | 17:20 |
| Clark[m] | Thank you for looking at it so quickly | 18:02 |
| corvus | okay, the explanation and emergency fix is in #zuul:opendev.org | 18:47 |
| corvus | we should merge that and get the schedulers running that | 18:47 |
| corvus | in the interim, zuul01 scheduler is running with a monkeypatch fix and zuul01 web is offline | 18:49 |
| corvus | bringing up zuul-web 01 with the new image; if it looks good, should be safe to restart scheduler 01, then both on 02 | 22:32 |
| corvus | secret version decoder ring: 73 is pre-upgrade, 114 is buggy image, 125 is fixed image | 22:32 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!