Tuesday, 2023-10-03

*** kopecmartin|off is now known as kopecmartin07:04
opendevreviewDaniel Blixt proposed zuul/zuul-jobs master: Make prepare-workspace-git work with unique scheme  https://review.opendev.org/c/zuul/zuul-jobs/+/89664007:56
opendevreviewDaniel Blixt proposed zuul/zuul-jobs master: Allow mirror push to delete current branch  https://review.opendev.org/c/zuul/zuul-jobs/+/76415207:57
opendevreviewMerged openstack/project-config master: Add Designate Bind K8S charm  https://review.opendev.org/c/openstack/project-config/+/89627212:18
opendevreviewJeremy Stanley proposed opendev/system-config master: Add mailing list for Nordix environment  https://review.opendev.org/c/opendev/system-config/+/89723414:44
clarkbarm image builds are all failing according to grafana15:03
clarkbalso opensuse seems to be going away upstream of us? we can probably wind down that distro in nodepool now15:03
clarkbthere is a new gitea 1.20.5 release (we are on 1.20.4) https://github.com/go-gitea/gitea/blob/v1.20.5/CHANGELOG.md nothign in there looks urgent for us16:06
clarkbbut we'll want to upgrade to that before the 1.21.x upgrade that eventually happens16:07
opendevreviewClark Boylan proposed opendev/system-config master: Upgrade to gitea 1.20.5  https://review.opendev.org/c/opendev/system-config/+/89724416:40
clarkbdiablo_rojo: fungi frickler I think ptgbot has crashed again18:31
fungiclarkb: it's still working in irc18:37
clarkbfungi: ya the webserver and ircbot are different processes. The ircbot didn't crash previously jsut the wbeserver which i think has occured again18:37
* tonyb is worried because his code merged and the bot crashed :(19:34
fungithe bot also crashed before that merged19:34
fungibut yes, getting some actual logging from the web process so we can capture it's last-gasp traceback would go a long way toward identifying the culprit19:35
fungiwithout looking closely, i'm going to guess that the web service is a fork so its stdout/stderr are not being captured by the init parent19:36
clarkbit wouldn't surprise me if it is a port contention thing. We can try restarting it with no code changes and see if it crashes the same way19:38
clarkbjust manually do tha down&& up -d or whatever ansible does for us19:38
tonybI looked at etherpad, the front page worked but I'm getting js errors on any actual pad.  I think that's due to the "bad" SSL cert19:45
fungiyou need to set etherpad.opendev.org to that address in your local /etc/hosts19:45
fungidon't try connecting by ip address, definitely won't work due to the api callbacks19:46
tonybI did that.19:47
fungijs errors may be the ones clarkb saw as well. clearing browser cache/storage/cookies might clear them up?19:47
clarkbtonyb: does that error go away if you use icognito mode? I think that was similar to what I got19:47
tonybThe error goes away with curl ;P19:48
clarkbinfra-root I ran docker-compose down on nb04 and it is taking a very long time to remove the container... hopefully it isn't completely broken. I have also started clearing /opt/dib_tmp contents19:48
tonybShoot it went away with a reload :/19:49
clarkbhttps://github.com/moby/moby/issues/22312 is the sort of thing I'm seeing on nb0419:49
clarkbtonyb: ya I think it is related to caching/cookies/sometemporal state19:49
clarkbI restarted the docker daemon and the container transitioned to a dead state19:51
clarkbbut trying to remove it is still looking like it will hit timeouts and fail19:51
tonybBased on my testings I'd say it (etherpad update) is weird19:51
clarkbya maybe we should avoid updating before the ptg as it is likely everyone will hit cache issues opening etherpad for the first time ina while if we do19:52
clarkbI was hoping I would be the only one but since you've reproduced it we may want to wait19:52
tonybYou're welcome ?19:54
clarkbwell better to have the info than be surprised later19:55
clarkbok the container eventually claened up19:56
clarkbonce dib_tmp is cleared out I'll reboot the server and restart containers19:56
JayFIs there an expected delay before a newly-configured ssh key in gerrit becomes usable?20:36
JayFTrying to determine if we're fighting cache or local config.20:36
JayFtrying to work in opendev/sandbox20:38
* JayF suspects potential git-in-wsl-using-wrong-ssh-key20:40
JayFwe figured it out; I think there was one too few buttons hit in the gerrit web ui20:47
diablo_rojofrickler: I booked a slot for the ptgbot on thursday of PTG week. Does that time work for you? 20:50
clarkbJayF: ya you have to hit the save button after adding the key20:52
clarkband no there isn't a delay20:52
JayFyeah we figured that out I think20:52
JayFBTW; opendev/sandbox was very helpful for this exercise; thank you21:23
fungiglad it's useful for something ;)21:24
opendevreviewMerged openstack/project-config master: Temporarily remove release docs semaphores  https://review.opendev.org/c/openstack/project-config/+/89694821:35
clarkbI went on a bike ride ande rm is still trying to clean up /opt on nb04 when I get back22:54
clarkbgood thing I didn't wait for it. I might have to reboot tomorrow at this rate though22:55
clarkbfungi: what time did the release team decide to start festivities tomorrow? I'm unlikely to wake up super early just given how my schedule has been recently but can probably check in around 1400 UTC to see if there is anything that needs attention23:00
opendevreviewClark Boylan proposed opendev/system-config master: Update our base container images  https://review.opendev.org/c/opendev/system-config/+/89727023:21

Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!