| opendevreview | Clark Boylan proposed opendev/engagement master: Add metric that counts changes left open at end of date range https://review.opendev.org/c/opendev/engagement/+/967253 | 00:12 |
|---|---|---|
| corvus | it looks like the zuul restarts completed without error. i'm going to start working on the zookeeper upgrade | 15:48 |
| corvus | zk02 appears to be the leader | 15:48 |
| corvus | i've pulled the 3.9 image on all 3 hosts so it's in the docker cache | 15:50 |
| corvus | i put all 3 hosts in the emergency file, and i don't see any deployment jobs running now | 15:53 |
| corvus | i'm going to down zk01; change the compose file; and up it. | 15:53 |
| corvus | actually... i'm going to down it, change the compose file, run an apt upgrade, and reboot, then up it. | 15:54 |
| corvus | seems happy. moving to zk03. | 15:58 |
| corvus | (the apt upgrades are, as expected, a noop) | 16:00 |
| corvus | zk03 seems happy | 16:01 |
| corvus | stopping the leader now, zk02 | 16:01 |
| corvus | approving the upgrade change | 16:03 |
| corvus | zk03 became the new leader | 16:05 |
| corvus | zk02 is back up and seems happy | 16:05 |
| corvus | the launchers (and the rest of zuul) appear to have recovered and are behaving normally | 16:09 |
| clarkb | corvus: should we go ahead and merge the change now? | 16:33 |
| clarkb | oh nevermind I think it is already in the gate so on its way | 16:34 |
| corvus | yeah, it's got to get through system-config-run-zuul which looks like it's expected to take another half hour | 16:34 |
| opendevreview | Merged opendev/system-config master: Upgrade ZooKeeper to 3.9 https://review.opendev.org/c/opendev/system-config/+/966612 | 17:10 |
| corvus | okay the deploy for that has run (presumably noop since they're in the emergency file) | 18:30 |
| corvus | i'm going to remove the zk servers from emergency now | 18:30 |
| corvus | that's done. the next deployment run that touches the zk servers should write those files for real and be a noop since the contents should be the same. | 18:31 |
| corvus | #status log upgraded zookeeper cluster to 3.9 | 18:31 |
| opendevstatus | corvus: finished logging | 18:31 |
| clarkb | thanks! | 19:13 |
Generated by irclog2html.py 4.0.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!