| opendevreview | OpenStack Proposal Bot proposed openstack/project-config master: Normalize projects.yaml https://review.opendev.org/c/openstack/project-config/+/957995 | 02:12 |
|---|---|---|
| fungi | just a reminder that i'm travelling for the next 1.5 weeks and won't be around much again until the 4th of september | 11:36 |
| Clark[m] | fungi: enjoy! Looks like the vldb entry for mirror.ubuntu is already locked preventing vos release of the reprepro updates. I'm going to assume that is due to the moves for now and will self correct once those complete | 13:32 |
| Clark[m] | Though I don't know if that is still in process or not | 13:33 |
| clarkb | I don't see any vos processes running on afs01.dfw at this point in time and `vos listvldb` there doesn't seem to show that mirror.ubuntu is locked | 16:33 |
| clarkb | then looking at the latest reprepro logs for mirror.ubuntu I think this started working again. I suspect the move finished and unlocked the volume and now things are happy again | 16:34 |
| clarkb | oh intersting the disk usage on afs02.dfw fell too so ya I think it was just hte move impacting that. I'm going to consider this happy for the moment and look at the next thing | 16:35 |
| *** melwitt_ is now known as jgwentworth | 16:43 | |
| *** jgwentworth is now known as melwitt | 16:44 | |
| opendevreview | Dmitriy Rabotyagov proposed openstack/diskimage-builder master: Add support for building Fedora 40 https://review.opendev.org/c/openstack/diskimage-builder/+/922109 | 17:06 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/diskimage-builder master: Add support for building Fedora 40 https://review.opendev.org/c/openstack/diskimage-builder/+/922109 | 17:12 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Upgrade etherpad to 2.5.0 https://review.opendev.org/c/opendev/system-config/+/956593 | 17:54 |
| opendevreview | Clark Boylan proposed opendev/system-config master: DNM force etherpad failure to hold node https://review.opendev.org/c/opendev/system-config/+/840972 | 17:55 |
| clarkb | tonyb: frickler: as part of setting up an autohold to test ^ I notice you both have a few autoholds in the openstack tenant. Should those be cleaned up or are we still using them? | 17:57 |
| frickler | clarkb: corvus: this is weird, I did have "sudo zuul-client autohold-delete --tenant openstack 0000000229" in my history. I reran the command now, it doesn't give any error, but the hold is still in place anyway. so some regression in zuul? (running the commands on zuul02 fwiw) | 18:00 |
| clarkb | I dont' expect a large attendance at the team meeting tomorrow but I plan on sending out an agenda and running throug it anyway as I think people do keep up with teh meeting notes asynchronously | 18:00 |
| clarkb | if you have any updates or items to discuss in the meeting tomorrow let me know or feel free toadd them to the agenda on the wiki directly | 18:01 |
| clarkb | frickler: I wonder if it is possible that those nodes were nodepool nodes and now that we've switched zuul is confused about how to clean that up? | 18:01 |
| frickler | the delete for 0000000232 went fine, however, but that didn't have a held node associated | 18:01 |
| frickler | that's possible, "Started at 2025-07-14 07:42:24" but no logs any longer to check. all those older holds might need some manual cleanup, then | 18:03 |
| frickler | also the held node still seems to be there, root@104.239.142.162 = root@np4c05860c99bc4 works | 18:04 |
| clarkb | that is a new style node name so not nodepool | 18:04 |
| clarkb | (its a sha rather than a counter) | 18:05 |
| clarkb | s/sha/uuid/ | 18:05 |
| corvus | i agree it's not being deleted; i tried through the web ui. we'll need to check the zuul-web logs to see if there's an error. | 18:07 |
| corvus | 2025-08-25 18:07:10,469 ERROR zuul.web: Exception: Failed to lock node <OpenstackProviderNode uuid=4c05860c99bc4fe9a011bb1e01544586, label=debian-trixie-8GB, state=hold, provider=opendev.org%2Fopendev%2Fzuul-providers/rax-dfw-main> | 18:09 |
| corvus | i have a guess: | 18:09 |
| corvus | we were leaking a bunch of node lock directories last month. i had to do manual cleanup of any dirs for unlocked nodes. i probably deleted that node's lock directory then. | 18:10 |
| corvus | i can re-create it in a bit, then we should be able to delete it. | 18:10 |
| corvus | done and done | 18:11 |
| clarkb | thanks! | 18:11 |
| corvus | that shouldn't be a problem for more recent autoholds, but if there were any others from around that time, it may affect them too. just lmk. | 18:12 |
| clarkb | will do | 18:13 |
| clarkb | I'm removing the rax-flex iad3 bootstrapping meeting topic. We're still at only 10 instances there (I just checked quotas) so we have one followup step to do there but that is largely dependent on cloudnull updating quotas (again no rush) | 20:13 |
| frickler | hmm, have the grafana dashboards for z-l rax* disappeared? also we could clean up all the nodepool ones by now? | 20:18 |
| clarkb | https://grafana.opendev.org/d/0172e0bb72/zuul-launcher3a-rackspace-flex?orgId=1&from=now-24h&to=now&timezone=utc&var-region=$__all and https://grafana.opendev.org/d/fd44466e7f/zuul-launcher3a-rackspace?orgId=1&from=now-24h&to=now&timezone=utc&var-region=$__all are still there and listed in the dashboards list for me | 20:19 |
| corvus | frickler: i'd like to keep the old one for months at least, maybe till end of year? | 20:19 |
| corvus | (i wouldn't object to keeping them for a full year -- until next july -- if folks wanted) | 20:20 |
| corvus | frickler: note that the list of dashboards is in a scrollable container inside the web page viewport, so even if you scroll to the bottom of the web page, you still need to scroll the interior container to see the dashboards at the bottom of the list | 20:21 |
| clarkb | I've done a first pass on meeting agenda updates. I'll probably send that out early today as its my children's turn for the annual eyeball inspection and I'll probably go to that appointment this afternoon | 20:21 |
| corvus | (talking about on this page: https://grafana.opendev.org/dashboards ) | 20:21 |
| clarkb | so let me know soonish if you have edits to make | 20:21 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Build trixie python base container images https://review.opendev.org/c/opendev/system-config/+/958480 | 20:39 |
Generated by irclog2html.py 4.0.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!