| corvus | i do not see the problem. :( | 00:04 |
|---|---|---|
| corvus | i'm going to restart and live patch in some extra debugging | 00:09 |
| fungi | i guess we need to wait for another recurrence? | 00:37 |
| corvus | yeah. i've restarted and re-enqueued and live-patched the extra debug log in | 00:56 |
| corvus | zuul01 is starting up now | 00:56 |
| corvus | i think i got it. it should be fixed in remote: https://review.opendev.org/c/zuul/zuul/+/816920 Fix change cache corruption on second scheduler start [NEW] | 03:15 |
| corvus | i think the thing to do now is to revert back to .4 and wait for that to land. it's a bit much to try to live-patch that one since it's part of the startup sequence. | 03:17 |
| corvus | restarted and re-enqueued | 03:45 |
| *** mazzy50988 is now known as mazzy5098 | 16:53 | |
| corvus | the fixes landed, so i'm going to restart so we can get some more data | 17:46 |
| corvus | starting up now; i'm going to run a quick errand then run the re-enqueue after i get back | 17:50 |
| fungi | thanks | 18:11 |
| corvus | re-enqueued and started zuul01 | 18:33 |
| corvus | done | 18:41 |
| corvus | we'll want to check back for errors in a few hours | 18:43 |
| fungi | yeah, i'll keep tabs on it | 18:53 |
| fungi | i've got a baseline count of the exceptions currently in the debug logs of both schedulers so i can see which ones increment | 18:55 |
| fungi | or if any new ones appear | 18:56 |
| corvus | no errors so far since zuul01 came online | 20:12 |
| fungi | yeah, i'm not seeing anything new on either scheduler still, just one "TypeError: 'NoneType' object is not subscriptable" in zuul.GithubEventProcessor at 20:30:57,375 utc | 20:57 |
| fungi | can't tell if it's pr or pr['head'] which is none there | 20:58 |
| fungi | guessing the latter though or it wouldn't have reached that line | 20:59 |
| fungi | that excpetion has been raised once more since the last time i scanned the log on zuul02 | 23:04 |
| opendevreview | Ian Wienand proposed opendev/system-config master: gerrit: mark file reviewed during testing https://review.opendev.org/c/opendev/system-config/+/816766 | 23:40 |
| ianw | i notice we didn't get a gerrit restart with the new theme plugin. if everything going well in a few hours re zuul i might do that just quickly before europe comes online | 23:43 |
| fungi | cool, i'll still be around for a while too | 23:53 |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!