| fungi | #status log Rebooted afs01.dfw after a block storage outage in its cloud provider at 21:30 UTC, and manually re-released all volumes to make sure none were left in a broken state | 00:00 |
|---|---|---|
| opendevstatus | fungi: finished logging | 00:03 |
| fungi | opendevstatus is clearly on island time, like me | 00:03 |
| mnaser | fungi: sorry my human nagios got you up and going :) | 00:48 |
| vlotorev[m] | <fungi> "also worth consideration: a..." <- It would be convenient if sandbox tenant woud have at least two projects to play round with cyclic dependencies :) | 01:08 |
| *** rpittau_ is now known as rpittau | 04:30 | |
| tkajinam | hmm it seems several jobs are stuck now ? https://zuul.opendev.org/t/openstack/status | 14:03 |
| corvus | yeah, looks like theres still a slow sql query. i'm going to do some emergency reverts in zuul to get us past this until we can fix it properly | 15:44 |
| corvus | i think we can continue forward with a small tweak (and doing so helps us get closer to the final sql queries we want anyway), so i have self-approved https://review.opendev.org/915203 and we can restart the schedulers/web when that's in. | 16:11 |
| corvus | in the interim, i've started a script that terminates slow sql queries to get things moving | 16:20 |
| corvus | okay, that merged, but because of my connection killing script, the promote jobs did not take effect. I'm going to manually pull the resulting image on the schedulers/web servers and restart | 17:30 |
| *** TheMaster is now known as Unit193 | 17:31 | |
| corvus | #status log restarted zuul-web and zuul-scheduler to pick up https://review.opendev.org/915203 | 17:42 |
| opendevstatus | corvus: finished logging | 17:42 |
| corvus | i've stopped my connection killing script, so we should be back to mostly normal behavior now. | 17:42 |
| fungi | thanks corvus!!! just now catching up but looks like things are starting to resume | 17:53 |
| fungi | interesting that some of the queries weren't even working that well under mysql instead of mariadb | 17:54 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!