| *** openstackgerrit has quit IRC | 00:36 | |
| *** DSpider has quit IRC | 01:16 | |
| *** nuclearg1 has quit IRC | 02:55 | |
| *** fressi has joined #opendev | 04:03 | |
| *** fressi has left #opendev | 04:03 | |
| *** ykarel has joined #opendev | 04:51 | |
| *** ykarel has quit IRC | 04:56 | |
| *** kenkenen1 has joined #opendev | 05:16 | |
| *** ykarel has joined #opendev | 05:51 | |
| *** ykarel has quit IRC | 05:52 | |
| frickler | infra-root: not sure I understood all the afs issues going on, but it also looks like debuntu mirrors were last released 5 days ago according to grafana | 07:05 |
|---|---|---|
| frickler | and gate backlog is huge, there seems to be a pretty high baseline of nodes being stuck in deletion. not sure if it is some nodepool issue or just nodes failing fast | 07:08 |
| frickler | so debuntu releases are failing because the volume is locked, someone needs to fix that, /me is lacking both experience and access to his kerberos credentials | 07:14 |
| frickler | "Could not lock the VLDB entry for the volume 536870949.\nVLDB: vldb entry is already locked\nError in vos release command." in reprepro logs | 07:15 |
| frickler | I can see lot's of these, all from rax-dfw on nl01: 2020-10-10 07:32:14,802 ERROR nodepool.NodeDeleter: Exception deleting instance f75c276e-72f3-40d3-bc0e-5fc28b761586 from rax-dfw: nodepool.exceptions.ServerDeleteException: Timeout waiting for server c7771dc8-845b-429e-a774-2ebc2f6f1130 deletion | 07:34 |
| frickler | also lots of failed launch attempts for ubuntu-xenial on airship-kna1 on nl02 | 07:37 |
| frickler | I'll try to take a deeper look later | 07:38 |
| *** DSpider has joined #opendev | 09:45 | |
| fungi | frickler: i'll take a look at the ubuntu volume lock, it's probably stale from when afs02.dfw hung on monday-ish | 11:29 |
| fungi | i've vos unlocked mirror.ubuntu and am initiating a new mirror run for it in a screen session now | 11:33 |
| fungi | it's well into releasing now | 11:48 |
| *** tosky has joined #opendev | 12:33 | |
| *** paladox has quit IRC | 13:32 | |
| *** paladox has joined #opendev | 13:35 | |
| fungi | but "This will be a full dump: previous release failed" so this will take a while. hopefully it doesn't exceed the ticket timeout... if it does i'll hold the reprepro lock and start a new vos release with -localauth on a fileserver | 13:46 |
| *** tosky has quit IRC | 14:53 | |
| fungi | "Released volume mirror.ubuntu successfully" | 15:53 |
| fungi | frickler: ^ we should be all clear there now | 15:53 |
| *** tosky has joined #opendev | 17:06 | |
| *** tosky has quit IRC | 18:01 | |
| clarkb | dirk: AJaeger: noting it here because I can't login to suse bugzilla even after creating an account for some reason. The tumbleweed lsb package requires /sbin/pidof but sysvinit-tools has been updated to stop providing it in favor of /usr/bin/pidof from procps. | 18:53 |
| clarkb | I resolved this by removing the lsb package since it appears to be a fake package to make other things happy anyway and nothing I have installed seems to dep on it | 18:54 |
| clarkb | oh I was just able to login | 18:54 |
| clarkb | I guess there si delay in account creation and bugzilla seeing it | 18:55 |
| fungi | they just wanted to make sure you weren't a mechanical turk | 18:55 |
| clarkb | in the end it was a pretty minor thing but the first option zypper gave me was to uninstall x11 :) | 19:01 |
| clarkb | that was not an option I wanted to take | 19:01 |
| AJaeger | clarkb: glad it worked out finally | 19:05 |
| fungi | you should be using wayland now, right? ;) | 19:05 |
| fungi | (or aqua) | 19:05 |
| AJaeger | I guess uninstalling x11 would deinstall wayland as well | 19:05 |
| *** roman_g has joined #opendev | 19:07 | |
| fungi | touché | 19:07 |
| *** roman_g has quit IRC | 20:08 | |
| *** roman_g has joined #opendev | 20:09 | |
| *** roman_g has quit IRC | 20:13 | |
| mwhahaha | can someone poke openstackgerrit | 20:24 |
| * fungi gets out his pokin' stick | 20:32 | |
| fungi | 2020-10-10 00:36:44 <-- openstackgerrit (~openstack@eavesdrop01.openstack.org) has quit (Ping timeout: 260 seconds) | 20:32 |
| fungi | yeah, that doesn't look so great | 20:33 |
| fungi | it | 20:33 |
| fungi | it's still receiving events from gerrit and thinks it's sending to freenode, but clearly no irc messages are making it through | 20:34 |
| mwhahaha | talking to the walk | 20:34 |
| mwhahaha | wall | 20:34 |
| fungi | #status log downed and upped gerritbot container on eavesdrop following irc timeout at 00:36:44 utc | 20:34 |
| openstackstatus | fungi: finished logging | 20:35 |
| fungi | at least statusbot is still working | 20:35 |
| fungi | minor saturday miracles | 20:37 |
| mwhahaha | :D | 20:37 |
| fungi | watching its log, but unsurprisingly not many people are pushing changes to gerrit right at this moment | 20:40 |
| fungi | maybe i need another drin | 20:40 |
| fungi | k | 20:40 |
| *** sgw has quit IRC | 21:14 | |
| *** sgw has joined #opendev | 21:14 | |
| *** qchris has quit IRC | 22:41 | |
| fungi | yeah, it's working again: http://eavesdrop.openstack.org/irclogs/%23openstack-requirements/%23openstack-requirements.2020-10-10.log.html#t2020-10-10T22:31:49-2 | 22:47 |
| fungi | all it took was a few drin | 22:47 |
| fungi | ks | 22:47 |
| *** tosky has joined #opendev | 22:51 | |
| *** qchris has joined #opendev | 22:53 | |
| *** tosky has quit IRC | 23:10 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!