Monday, 2020-12-21

*** tosky has quit IRC00:12
*** sshnaidm is now known as sshnaidm|afk00:22
*** Alex_Gaynor has quit IRC00:46
*** Alex_Gaynor has joined #opendev00:47
*** elod_pto has quit IRC00:57
*** tbarron has quit IRC00:57
*** valleedelisle has quit IRC00:57
*** mhu has quit IRC00:57
*** stephenfin has quit IRC00:57
*** openstackgerrit has quit IRC01:00
*** DSpider has quit IRC01:10
*** elod_pto has joined #opendev01:17
*** tbarron has joined #opendev01:17
*** valleedelisle has joined #opendev01:17
*** mhu has joined #opendev01:17
*** stephenfin has joined #opendev01:17
*** auristor has quit IRC01:27
*** auristor has joined #opendev01:32
*** tkajinam has quit IRC02:35
*** tkajinam has joined #opendev02:35
*** brinzhang has joined #opendev02:41
*** brinzhang_ has joined #opendev04:12
*** brinzhang has quit IRC04:15
*** ianw is now known as ianw_pto04:22
*** ykarel has joined #opendev04:44
*** bhagyashris is now known as bhagyashri|ruck05:27
*** bhagyashri|ruck is now known as bhagyashris|ruck05:28
*** marios has joined #opendev06:04
*** hamalq has joined #opendev06:50
*** hamalq has quit IRC06:55
*** ykarel has quit IRC07:21
*** slaweq has joined #opendev07:45
*** ykarel has joined #opendev07:57
*** ralonsoh has joined #opendev08:02
*** marios has quit IRC08:03
*** rpittau|afk is now known as rpittau08:07
*** ykarel is now known as ykarel|lunch08:25
*** ykarel|lunch has quit IRC08:25
*** cloudnull has quit IRC08:53
*** cloudnull6 has joined #opendev08:53
*** lpetrut has joined #opendev08:54
*** ykarel has joined #opendev09:03
*** ykarel has quit IRC09:10
*** hashar has joined #opendev09:18
*** hamalq has joined #opendev09:28
*** ykarel has joined #opendev09:29
*** hamalq_ has joined #opendev09:29
*** ykarel_ has joined #opendev09:33
*** hamalq has quit IRC09:33
*** fabv has joined #opendev09:34
*** hamalq_ has quit IRC09:34
*** ykarel has quit IRC09:36
*** ykarel_ is now known as ykarel09:36
fabvhello, could someone review this please?09:36
*** marios has joined #opendev09:49
*** tosky has joined #opendev10:03
*** brinzhang_ has quit IRC10:16
*** brinzhang_ has joined #opendev10:17
*** JayF has quit IRC10:34
*** JayF has joined #opendev10:38
*** ykarel_ has joined #opendev10:55
*** ykarel has quit IRC10:58
*** sshnaidm|afk is now known as sshnaidm11:17
*** hamalq has joined #opendev11:30
*** dviroel has joined #opendev11:32
*** hamalq has quit IRC11:35
*** DSpider has joined #opendev11:35
*** ykarel_ is now known as ykarel11:42
*** larainema has quit IRC11:46
*** dtantsur|afk is now known as dtantsur11:47
*** hamalq has joined #opendev11:58
*** fabv has quit IRC12:00
*** hamalq has quit IRC12:03
*** iurygregory has joined #opendev13:16
danpawlikfungi: hey, is it possible to merge that before next year: ?13:25
fungidanpawlik: i'll need to check the available quota on that volume but can take a look13:27
danpawlikfungi thank you13:27
fungi indicates mirror.fedora has plenty of room, excellent13:28
*** marios has quit IRC13:41
*** ykarel_ has joined #opendev13:52
*** brinzhang_ has quit IRC13:54
*** ykarel has quit IRC13:54
*** brinzhang_ has joined #opendev13:55
*** ykarel_ is now known as ykarel13:55
*** cloudnull6 is now known as kecarter13:58
*** hamalq has joined #opendev13:59
*** kecarter is now known as cloudnull14:03
*** hamalq has quit IRC14:04
*** marios has joined #opendev14:39
*** lpetrut has quit IRC14:51
frickler#status log restarted gerritbot on eavesdrop.o.o which seemed to have lost connections14:59
openstackstatusfrickler: finished logging14:59
fungiyeah, a couple hours ago i pushed a change which should have been reported in two channels but only one got notified, which was sort of strange15:30
fungithanks for the restart!15:30
*** ykarel has quit IRC15:36
fungilooks like lists.o.o is broken. apache returns nothing, can't ssh into it... i thought there were surprisingly few ml posts when i woke up, guess they're all queued by senders. will work on getting the server back in shape now15:48
corvusfungi: ow; lmk if you need a hand15:50
fungithanks, just checking the emergency console in rax first15:52
clarkbI'm slowly catching up too15:53
funginothing on the console, and no response to carriage return. will try to use the ctrl+alt+del button it provides15:53
fungiif this doesn't work, i'll reboot via api15:53
fungiyeah, nothing. soft reboot triggered via api now...15:55
fungidoesn't seem to have taken effect. will do --hard15:58
clarkb(that is "normal")15:58
*** hamalq has joined #opendev16:00
*** hamalq has quit IRC16:04
fungilists.o.o seems to be online again for ~7 minutes now16:07
fungitaking a look to make sure it seems sane16:07
fungii'm able to process the moderation queues16:08
fungimessages are coming through from openstack-discuss already16:12
clarkbI can confirm I see messages arriving16:13
fungijudging from cacti the server died between 04:05 and 04:10 utc, i think16:13
fungiunfortunately i stopped being able to use the graph zoom in cacti some months back (maybe when the server was rebuilt? maybe something with my browser)16:14
clarkbI can confirm that as well16:15
fungi#status log rebooted at 16:00z to remedy complete hang around 04:10z today16:15
openstackstatusfungi: finished logging16:15
fungianyway, i think that's back on track16:16
fungiunrelated, rackspace opened a support ticket on saturday to inform us of a block storage outage impacting around 05:45 utc that day... i haven't found time to dig into it yet16:18
clarkbI can probably take a look at the nodepool builder a bti later today just to check it is happy16:19
fungidmesg indicates i/o errors on xvdc starting at 2020-12-19 02:56:06 immediately following hung task timeouts at 02:41:1016:22
fungiamd /opt seems to now be mounted read-only16:23
clarkbfungi: rebooting should be fine for those servers (rather than remounting)16:23
clarkbwe get a fsck automatically if we do that?16:23
clarkb(though maybe a fsck is not desireable?)16:24
fungiyeah, i'm thinking it will since the fs will be marked dirty16:24
fungiwe could disable fsck at boot for /opt in fstab, but would also need to disable nodepool services on it if we want it to come up idle so we can manually fsck or mkfs it16:25
fungiin my opinion we should just reboot and let the chips fall where they may16:26
fungidoing that now16:26
fungiDec 21 16:27:29 nb01 systemd-fsck[578]: main01: recovering journal16:29
fungiDec 21 16:27:29 nb01 systemd-fsck[578]: main01: clean, 379798/67108864 files, 146014364/268434432 blocks16:29
fungiDec 21 16:27:29 nb01 systemd[1]: Mounted /opt.16:30
fungiso should be good to go16:30
fungi#status log rebooted nb01.o.o to recover from a block device outage around 2020-12-19 03:00z16:31
openstackstatusfungi: finished logging16:31
*** hamalq has joined #opendev17:00
*** marios has quit IRC17:01
*** ralonsoh has quit IRC17:05
*** ralonsoh has joined #opendev17:05
*** openstackgerrit has joined #opendev17:12
openstackgerritAlbin Vass proposed zuul/zuul-jobs master: Add configuration to make logs public
*** rpittau is now known as rpittau|afk17:13
*** hamalq has quit IRC17:14
*** hamalq has joined #opendev17:14
openstackgerritMerged ttygroup/gertty master: Handle gerrit 3.x style internal urls
openstackgerritMerged ttygroup/gertty master: Add WIP support
*** hamalq has quit IRC17:17
*** hamalq has joined #opendev17:17
*** ralonsoh_ has joined #opendev17:24
*** ralonsoh has quit IRC17:25
*** fdegir has quit IRC17:25
*** fdegir has joined #opendev17:25
openstackgerritAlbin Vass proposed zuul/zuul-jobs master: upload-artifactory: no_log upload task
openstackgerritAlbin Vass proposed zuul/zuul-jobs master: upload-artifactory: no_log upload task
*** stzsch has quit IRC17:35
*** tristanC has quit IRC17:44
*** tristanC has joined #opendev17:46
*** hashar has quit IRC18:00
*** ralonsoh_ has quit IRC18:01
*** ysandeep is now known as ysandeep|away18:05
*** dtantsur is now known as dtantsur|afk18:15
*** stzsch has joined #opendev18:25
*** yoctozepto has quit IRC19:55
*** yoctozepto has joined #opendev19:55
*** tkajinam_ has joined #opendev21:31
*** ShadowJonathan has quit IRC21:32
*** TheJulia has quit IRC21:33
*** ShadowJonathan has joined #opendev21:34
*** hamalq has quit IRC21:34
*** TheJulia has joined #opendev21:34
*** tkajinam has quit IRC21:34
*** hamalq has joined #opendev21:34
*** rpittau|afk has quit IRC23:21
*** johnsom has quit IRC23:21
*** TheJulia has quit IRC23:22
*** tosky has quit IRC23:41
*** redrobot has quit IRC23:43

Generated by 2.17.2 by Marius Gedminas - find it at!