| tonyb | Clark[m]: I don't know if this is normal. The zuul_reboot is still on ze01 and seems to stuck in a loop trying to stop the last two jobs but unable to find the workers? https://paste.opendev.org/show/bcaTNgofTKeMHCmfZXMi/ | 01:09 |
|---|---|---|
| tonyb | Oh actually it's not the same event/build | 01:12 |
| tonyb | I have convinced myself it's perfectly normal. The 2 jobs running on ze01 are long tempest jobs that started 23:58 and 23:59 | 01:27 |
| tonyb | yeah it's moving now. | 02:08 |
| corvus | zl01 is stuck waiting to shutdown because one of its upload worker threads is stuck on an http write call. i don't know which, we might be able to figure that out from the logs. but we'll need some changes to handle this differently. i'll kill it so the restart can continue. | 13:30 |
| fungi | a quick check of https://zuul.opendev.org/components indicates it's working on the launchers now | 14:39 |
| fungi | executors, mergers, and zl01 are upgraded, zl02 isn't listed, schedulers aren't upgraded yet | 14:40 |
| fungi | 965447 promoted between the ze03 and ze04 upgrades, hopefully having some without it won't be a problem | 14:42 |
| Clark[m] | I wonder if zl02 is crashing as a side effect of the other restarts so that when we get to it it isn't running and we fail on the bug I pushed a fix for yesterday | 14:53 |
| corvus | fungi: i checked before i approved those changes; 447 won't affect the executors. | 15:00 |
| corvus | zl02 looks to be in the same stuck upload situation as zl01 | 15:02 |
| corvus | i dislodged it as well; the playbook is continuing | 15:03 |
| corvus | all done now | 15:22 |
| fungi | confirmed, everything looks upgraded now | 16:10 |
| fungi | thanks corvus! | 16:10 |
Generated by irclog2html.py 4.0.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!