Saturday, 2023-07-29

fungi#status log Some changes and pipelines have been stuck for the past 8 hours due to an upgrade-related Zuul bug, fix is in flight at
opendevstatusfungi: finished logging15:53
corvusfungi: i think we can restart the schedulers now with that fix.  you want to do that or shall i?16:31
fungicorvus: i can. how have you been doing it in the past? edited restart playbook that limits to just the scheduler servers?16:35
corvusfungi: for this -- i would probably just log into each of the 2 servers in turn and down/pull/up each of the schedulers -- there's only two :)16:36
fungifair, i can do that now16:37
fungii'm downing the containers on zuul0116:38
fungino need to graceful stop them first, right?16:38
corvusi think the down will send them a signal to stop16:38
fungidowned and pulling in progress now16:40
fungizuuld processes are still running16:41
fungii guess i need to wait until they clean up?16:41
corvusyeah, should be relatively quick16:41 latest 9f9a55134eb0 55 minutes ago 719MB16:42
corvusfungi: oh it is also probably worth doing the same for zuul-web; this change might affect that16:42
fungii did all the containers on zuul0116:42
fungia full docker-compose down16:43
corvusfungi: zuul-web and zuul-fingergw have a separate docker-compose from zuul-scheduler16:43
fungiaha, right16:44
fungiforgot we had separate compose files16:44
fungiweb and fingergw are down and have new images pulled too16:45
fungiand no zuuld processes running, so i'll up -d them16:45
fungipresumably i need to tail the debug logs to see when they're done starting up before i take down the containers on zuul02?16:46
corvusfungi: or check till it's all green16:47
fungioh, i didn't realize that was quite so real-time. awesome16:47
corvusyep; it doesn't auto-refresh, so you'll have to ctrl-r but it is real-time at time of render16:49
fungiand all green. starting on 02 now16:50
fungiopenstack tenant's gate pipeline cleared out and stuff has started moving again in check as well16:54
corvusfungi: looks like things are unstuck now?16:54
fungiokay, both schedulers are back to running, now on 8.3.2.dev73 596da2d9316:56
fungi#status log Zuul schedulers have been updated to fixed images and everything's moving normally again16:56
opendevstatusfungi: finished logging16:56
fungithanks for all the help corvus!16:56
corvusyw :)17:02
corvusi'm going to restart all of zuul now to pick up 88331818:34
fungithat looks like an efficiency improvement we've been wanting for years19:16
corvusyeah i hope to do more with it later, but hopefully that's a good start19:41
fungiseems like it may result in a pretty good throughput improvement for dependent pipelines on rainy days19:42
corvus#status log restarted all of zuul on 6c0ffe565f1d0025ccee08a697cc73b4594942e519:54
opendevstatuscorvus: finished logging19:54
* fungi rubs his hands together19:54
corvusnow we just need some jobs to run :)19:55

Generated by 2.17.3 by Marius Gedminas - find it at!