Saturday, 2020-10-10

*** openstackgerrit has quit IRC00:36
*** DSpider has quit IRC01:16
*** nuclearg1 has quit IRC02:55
*** fressi has joined #opendev04:03
*** fressi has left #opendev04:03
*** ykarel has joined #opendev04:51
*** ykarel has quit IRC04:56
*** kenkenen1 has joined #opendev05:16
*** ykarel has joined #opendev05:51
*** ykarel has quit IRC05:52
fricklerinfra-root: not sure I understood all the afs issues going on, but it also looks like debuntu mirrors were last released 5 days ago according to grafana07:05
fricklerand gate backlog is huge, there seems to be a pretty high baseline of nodes being stuck in deletion. not sure if it is some nodepool issue or just nodes failing fast07:08
fricklerso debuntu releases are failing because the volume is locked, someone needs to fix that, /me is lacking both experience and access to his kerberos credentials07:14
frickler"Could not lock the VLDB entry for the volume 536870949.\nVLDB: vldb entry is already locked\nError in vos release command." in reprepro logs07:15
fricklerI can see lot's of these, all from rax-dfw on nl01: 2020-10-10 07:32:14,802 ERROR nodepool.NodeDeleter: Exception deleting instance f75c276e-72f3-40d3-bc0e-5fc28b761586 from rax-dfw: nodepool.exceptions.ServerDeleteException: Timeout waiting for server c7771dc8-845b-429e-a774-2ebc2f6f1130 deletion07:34
frickleralso lots of failed launch attempts for ubuntu-xenial on airship-kna1 on nl0207:37
fricklerI'll try to take a deeper look later07:38
*** DSpider has joined #opendev09:45
fungifrickler: i'll take a look at the ubuntu volume lock, it's probably stale from when afs02.dfw hung on monday-ish11:29
fungii've vos unlocked mirror.ubuntu and am initiating a new mirror run for it in a screen session now11:33
fungiit's well into releasing now11:48
*** tosky has joined #opendev12:33
*** paladox has quit IRC13:32
*** paladox has joined #opendev13:35
fungibut "This will be a full dump: previous release failed" so this will take a while. hopefully it doesn't exceed the ticket timeout... if it does i'll hold the reprepro lock and start a new vos release with -localauth on a fileserver13:46
*** tosky has quit IRC14:53
fungi"Released volume mirror.ubuntu successfully"15:53
fungifrickler: ^ we should be all clear there now15:53
*** tosky has joined #opendev17:06
*** tosky has quit IRC18:01
clarkbdirk: AJaeger: noting it here because I can't login to suse bugzilla even after creating an account for some reason. The tumbleweed lsb package requires /sbin/pidof but sysvinit-tools has been updated to stop providing it in favor of /usr/bin/pidof from procps.18:53
clarkbI resolved this by removing the lsb package since it appears to be a fake package to make other things happy anyway and nothing I have installed seems to dep on it18:54
clarkboh I was just able to login18:54
clarkbI guess there si delay in account creation and bugzilla seeing it18:55
fungithey just wanted to make sure you weren't a mechanical turk18:55
clarkbin the end it was a pretty minor thing but the first option zypper gave me was to uninstall x11 :)19:01
clarkbthat was not an option I wanted to take19:01
AJaegerclarkb: glad it worked out finally19:05
fungiyou should be using wayland now, right? ;)19:05
fungi(or aqua)19:05
AJaegerI guess uninstalling x11 would deinstall wayland as well19:05
*** roman_g has joined #opendev19:07
*** roman_g has quit IRC20:08
*** roman_g has joined #opendev20:09
*** roman_g has quit IRC20:13
mwhahahacan someone poke openstackgerrit20:24
* fungi gets out his pokin' stick20:32
fungi2020-10-10 00:36:44     <--     openstackgerrit ( has quit (Ping timeout: 260 seconds)20:32
fungiyeah, that doesn't look so great20:33
fungiit's still receiving events from gerrit and thinks it's sending to freenode, but clearly no irc messages are making it through20:34
mwhahahatalking to the walk20:34
fungi#status log downed and upped gerritbot container on eavesdrop following irc timeout at 00:36:44 utc20:34
openstackstatusfungi: finished logging20:35
fungiat least statusbot is still working20:35
fungiminor saturday miracles20:37
fungiwatching its log, but unsurprisingly not many people are pushing changes to gerrit right at this moment20:40
fungimaybe i need another drin20:40
*** sgw has quit IRC21:14
*** sgw has joined #opendev21:14
*** qchris has quit IRC22:41
fungiyeah, it's working again:
fungiall it took was a few drin22:47
*** tosky has joined #opendev22:51
*** qchris has joined #opendev22:53
*** tosky has quit IRC23:10

Generated by 2.17.2 by Marius Gedminas - find it at!