Monday, 2022-02-28

ianw#status log redirected old jjb at docs.o.o/infra/jenkins-job-builder to RTD site; moved old content to ./attic/* and added a .htaccess and README in the original dir.  see
opendevstatusianw: finished logging00:11
ianwAdamR: ^ i replied on list, but let me know if issues00:11
opendevreviewMerged openstack/project-config master: Add Zuul performance metrics dashboard
fungiianw: if you're good with 830912 i can do the extra steps it needs in a gerrit restart once it's merged/published01:39
ianwfungi: oh, sorry i thought it was in!02:10
ianwi can clean those up when it merges too if you like02:10
fungioh, if you're up for it that'd be great. it's getting fairly late here so i'm less sure i'll be awake once the images update02:11
fungiper the commit message, the logfiles have to be removed/moved while the service is offline02:12
ianwwe moved the periodic jobs forward, i guess most people don't care but it does slow things down in the pacific afternoon02:25
ianwi feel like two or so hours from now might be even quieter02:26
ianwa constant battle with daylight savings times02:26
fungisounds fine to me. i'll try to check on it first thing when i wake up in that case02:28
opendevreviewMerged opendev/system-config master: Remove Gerrit's JVM GC logs
ianwstill note seeing ^ on hub.docker03:44
opendevreviewIan Wienand proposed zuul/zuul-jobs master: ensure-pip: fix typo in ensure_pip_virtualenv_command documentation
ianwoh doh; 830912 doesn't need to push an image.  i can just quickly restart per the instructions now05:07
ianw#status log stopped gerrit, moved jvm log files to ~gerrit2/tmp/jvm-logs, restarted gerrit to apply 83091205:11
opendevstatusianw: finished logging05:11
ianwnot seeing the errors in logs05:14
*** ysandeep|out is now known as ysandeep05:59
*** ykarel_ is now known as ykarel06:23
*** luigi is now known as luigi-training06:39
*** luigi-training is now known as luigi06:39
*** ysandeep is now known as ysandeep|lunch07:18
*** amoralej|off is now known as amoralej07:24
*** bhagyashris_ is now known as bhagyashris|ruck07:56
*** jpena|off is now known as jpena09:42
*** ysandeep|lunch is now known as ysandeep09:53
*** rlandy is now known as rlandy|ruck11:15
*** sshnaidm|off is now known as sshnaidm11:38
fungiyeah, sorry, i meant deployed, since it was a change to the docker-compose config13:13
fungiand yes it looks like the jvm_gc.log has not returned either13:19
fungiand gerrit is still running steadily as far as I can see13:20
opendevreviewMaksim Malchuk proposed openstack/diskimage-builder master: Correctly create DIB_ENV variable and dib_environment file
*** ysandeep is now known as ysandeep|dinner14:43
*** rlandy|ruck is now known as rlandy|ruck|mtg14:54
*** ysandeep|dinner is now known as ysandeep15:12
diablo_rojoLooks like ethercalc is maybe down?15:14
diablo_rojoWas going to look at the PTG ethercalc:
fungithanks, checking15:28
fungiError: Can't set headers after they are sent.15:31
fungii don't think that's the issue we've seen in the past15:32
fungi#status log Started the Ethercalc service after it crashed at 11:29:07 UTC15:34
opendevstatusfungi: finished logging15:34
fungidiablo_rojo: ^15:34
fungilooks like it was down for roughly 4 hours15:34
fungimaybe that is the one we keep running into after all15:37
clarkbfungi: ianw: so the mergability chcek change is in as is the jvm gc log cleanup to remove the errors from our logs? Did the apache update manage to get in somehow? I was going to look at the zuul registry more closely today once caught up generally16:17
fungiclarkb: the first two yes, the apache vhost config update to block access to gitiles has not merged yet since it's still blocked on the registry bug, but also not terribly urgent16:18
clarkbfungi: ok, I don't know if you saw but the testinfra test is why we rebuild the image. We could either stop running testinfra on image updates (I think taht is a bad idea since new images should get testinfra'd) or split the testing from the change if we like. Based on what I found Friday I'm not so sure the registry will be a quick and easy fix16:19
fungiyeah, i don't mind splitting the test to a second change, but the test is testing the adjusted config works, so i'm also fine with trying to address the registry bug first16:21
clarkbok. Once I'ev settled in a bit and caught up on some reviews and other stuff (gitea fixed our bug I need to check if that is in 1.16.2 for example) I'll look at the registry and see if I can spy why two uploads of the same object is problematic in the registry16:22
clarkbLooks like the gitea bugfix is on 1.16.3. I'm happy to wait a bit logner and upgrade to that version if it shows up soon. Looks like it has a few other bugfixes too16:30
*** rlandy|ruck|mtg is now known as rlandy|ruck16:31
fungiyeah, that seems prudent and saves us some work16:35
opendevreviewJames E. Blair proposed openstack/project-config master: Correct typo in zuul-performance dashboard
corvusclarkb: fungi ^ that's a trivial fix if you have a sec16:55
*** ysandeep is now known as ysandeep|out16:59
clarkbI think I understand the zuul-registry bug17:03
clarkbthe paste I made last week was on the right track but for the wrong object. Bsaically what happens is if a second client pushes the same object while the other one is writing the object we return to the client and say it is done. It immediately does a HEAD against the object. Before my atomic move change this could be a short read depending on how much had been written. After the17:05
clarkbatomic move it was always a 40417:05
clarkbI'll work on a fix. Though I'm not sure what the correct fix is yet. I think with the atomic moves we can stop returning early and just have each client move the file in place atomically and avoid the race entirely17:05
clarkbbut I need to update my change to not use a consistent tmpname17:06
opendevreviewMerged openstack/project-config master: Correct typo in zuul-performance dashboard
*** jpena is now known as jpena|off17:31
clarkb here's a first draft of the zuul-registry fixup17:36
clarkbI've managed to skip breakfast. I'm going to go eat then likely dig into reviews. Looks like there are a few in zuul land that need attention17:57
diablo_rojofungi, thanks for looking into it! Can confirm its back up now. 17:59
opendevreviewClark Boylan proposed zuul/zuul-jobs master: DNM testing depends-on
*** artom__ is now known as artom19:02
opendevreviewMerged openstack/diskimage-builder master: Revert "Revert "Detect boot and EFI partitions in extract-image""
opendevreviewClark Boylan proposed zuul/zuul-jobs master: DNM testing depends-on
*** rlandy|ruck is now known as rlandy|ruck|biab22:51
*** rlandy|ruck|biab is now known as rlandy|ruck23:13
ianwis it just me or is element/matrix not happy?23:14
clarkbianw: seems to be working for me23:24
clarkbthough I've not tried to send messages for a bit23:24
clarkbianw: once is in we should expect happy rocky?23:38
ianwclarkb: oh, that hit so many transient issues yesteday :/23:39
ianwif it's gone in, i think we can expect one less problem with rocky -- and hopefully it only has one problem remaining :)23:39
clarkbno the nodepool update hasn't happened yet. But sounds like that is the next step at least23:40
clarkbI'm putting together a meeting agenda tomorrow and that was one of the thigns to catch up on. Please update the wiki if I've missed anything important for tomorrow23:41
clarkbI'll try to send that out in about 20 minutes or so23:42
fungii'm not aware of anything that needs adding23:42
clarkbianw: there asn't any non rocky fallout from the glean release was there?23:43
clarkblooks like no recent glean changes at least if s23:43
ianwi don't believe so ... i watches some builds and haven't seen reports of failures to boot, etc.23:44
clarkbthanks! I meant to followup on that and got sniped by the registry23:44

Generated by 2.17.3 by Marius Gedminas - find it at!