Sunday, 2021-11-07

corvusi do not see the problem.  :(00:04
corvusi'm going to restart and live patch in some extra debugging00:09
fungii guess we need to wait for another recurrence?00:37
corvusyeah.  i've restarted and re-enqueued and live-patched the extra debug log in00:56
corvuszuul01 is starting up now00:56
corvusi think i got it.  it should be fixed in remote: Fix change cache corruption on second scheduler start [NEW]        03:15
corvusi think the thing to do now is to revert back to .4 and wait for that to land.  it's a bit much to try to live-patch that one since it's part of the startup sequence.03:17
corvusrestarted and re-enqueued03:45
*** mazzy50988 is now known as mazzy509816:53
corvusthe fixes landed, so i'm going to restart so we can get some more data17:46
corvusstarting up now; i'm going to run a quick errand then run the re-enqueue after i get back17:50
corvusre-enqueued and started zuul0118:33
corvuswe'll want to check back for errors in a few hours18:43
fungiyeah, i'll keep tabs on it18:53
fungii've got a baseline count of the exceptions currently in the debug logs of both schedulers so i can see which ones increment18:55
fungior if any new ones appear18:56
corvusno errors so far since zuul01 came online20:12
fungiyeah, i'm not seeing anything new on either scheduler still, just one "TypeError: 'NoneType' object is not subscriptable" in zuul.GithubEventProcessor at 20:30:57,375 utc20:57
fungican't tell if it's pr or pr['head'] which is none there20:58
fungiguessing the latter though or it wouldn't have reached that line20:59
fungithat excpetion has been raised once more since the last time i scanned the log on zuul0223:04
opendevreviewIan Wienand proposed opendev/system-config master: gerrit: mark file reviewed during testing
ianwi notice we didn't get a gerrit restart with the new theme plugin.  if everything going well in a few hours re zuul i might do that just quickly before europe comes online23:43
fungicool, i'll still be around for a while too23:53

Generated by 2.17.2 by Marius Gedminas - find it at!