| @abelur:matrix.org | @seongsoocho Clark to help move the i18n weblate migration along - 961499 has been idle since early Aug and its dependency merged on the 22nd. Left some comments on the review on 961499 | 04:15 |
|---|---|---|
| -@gerrit:opendev.org- Dr. Jens Harbott proposed: [openstack/project-config] 1006511: release-tools: fix typo and line length https://review.opendev.org/c/openstack/project-config/+/1006511 | 12:26 | |
| -@gerrit:opendev.org- Zuul merged on behalf of Dr. Jens Harbott: [openstack/project-config] 1006511: release-tools: fix typo and line length https://review.opendev.org/c/openstack/project-config/+/1006511 | 12:40 | |
| @fungicide:matrix.org | i'm officially back as of now, but it's going to take me some time to catch up on discussions from the past week so please bear with me in the meantime | 12:57 |
| @mnasiadka:matrix.org | So, I'm seeing a lot of connection reset by peer when connecting to Ansible Galaxy in VexxHost's ca-ymq-1 region (in Kolla/Kolla-Ansible jobs) | 13:05 |
| @mnasiadka:matrix.org | e.g. https://zuul.opendev.org/t/openstack/build/caba73e17e404230ab2c97db93d8bc23 | 13:06 |
| @mnasiadka:matrix.org | fungi: happy to clear out Debian Contents-* and do the same for Ubuntu - unless you want to beat me to it ;-) | 13:13 |
| @fungicide:matrix.org | mnasiadka: any idea if the intermittent ipv6 internal routing problem in ca-ymq-1 ever got resolved? the 2604:e100:1::1 router there is back to claiming it can't find the gerrit server when i try to `ping6 review.opendev.org` from random places on the internet | 13:13 |
| @mnasiadka:matrix.org | No, it never got - I raised a ticket with them to debug some ipv4 connectivity problems from Europe to Gerrit, but they mainly break on twelve99.net (a transit provider) | 13:14 |
| @mnasiadka:matrix.org | So maybe we need to raise one about the ipv6 connectivity | 13:14 |
| @fungicide:matrix.org | mnasiadka: i added the contents-*.gz followup to my todo list, but only so it's not forgotten, i'm unlikely to get to it before october | 13:14 |
| @mnasiadka:matrix.org | https://support.vexxhost.com/hc/en-us/requests/new | 13:15 |
| @fungicide:matrix.org | the reason i bring up the internal ipv6 routing oddities is that what we see with gerrit may also be a symptom of whatever is breaking your ansible galaxy access from there in zuul jobs (ig galaxy pyblishes v6 rrs) | 13:16 |
| @mnasiadka:matrix.org | Yes, I did run some tests from the mirror in ca-ymq-1 and I get connection reset by peer via ipv6 from time to time | 13:17 |
| @fungicide:matrix.org | previously, the gerrit v6 access issues would clear up temporarily if i ssh into it and `ping 2604:e100:1::1` from there | 13:17 |
| @mnasiadka:matrix.org | Yeah, after doing that - review.opendev.org responds to ping6 | 13:18 |
| @harbott.osism.tech:regio.chat | so - we should keep that running in a screen session for now? | 13:19 |
| @fungicide:matrix.org | or stick something like `ping -qc3 2604:e100:1::1` in a every-minute cronjob | 13:20 |
| @fungicide:matrix.org | that may work around the gerrit v6 issues, but probably won't solve e.g. jobs having trouble reaching ansible galaxy | 13:22 |
| @mnasiadka:matrix.org | Seems to be working now from the mirror after issuing that ping (on mirror03) - wonder if we're using mirror as the proxy on that ansible-galaxy run | 13:24 |
| @mnasiadka:matrix.org | I'd say it would be better to raise a support ticket than run ping :) | 13:37 |
| @fungicide:matrix.org | agreed | 14:00 |
| @fungicide:matrix.org | which is why i called it a workaround. something we might do temporarily while vexxhost is looking into the problem | 14:00 |
| @clarkb:matrix.org | fungi: we never statically configured review03's ipv6 address like we did with review02 iirc. It just never seemed necessary as the RAs seemed to work. But maybe that changed and we need to statically configure ipv6 on these hosts because their upstream routers are not properly sending config downstream? | 14:45 |
| @clarkb:matrix.org | Have we checked the local `ip addr` and `ip route` info when things are unhappy to see if this is a reoccurrence of missing ipv6 configuration? | 14:46 |
| @fungicide:matrix.org | but it sounds like there may also be a similar problem with the mirror server there, as well as with test nodes getting launched there by zuul | 14:46 |
| @fungicide:matrix.org | i did check ip address and route info on the gerrit server week before last when i was looking into it and everything looked normal, though mnaser expressed surprise that the server was only seeing one v6 default gateway. i wonder if advertisements from one of the routers there isn't making it to servers for some reason | 14:47 |
| @fungicide:matrix.org | and similarly, the router doesn't see the servers in its neighbor table | 14:48 |
| @fungicide:matrix.org | that would at least be consistent with the router suddenly being able to forward packets to the server after getting pinged from it | 14:49 |
| @mnasiadka:matrix.org | Ok, we're getting weird RA packets from other hosts than the default router | 15:05 |
| @mnasiadka:matrix.org | Doesn't Neutron do RA anti-spoofing based on port_security - which we don't use? | 15:07 |
| @fungicide:matrix.org | it depends on what bridge layer is in use i think, but yes neutron essentially doesn't implement separate anti-spoofing measures by default (i.e. manually tweaking bridge filters) and expects port security to take care of it | 15:09 |
| @fungicide:matrix.org | at least that's my (non-operator, non-neutron-dev) recollection | 15:09 |
| @mnasiadka:matrix.org | Ok, but our ports in ca-ymq-1 use port_security, so that should be fine | 15:10 |
| @clarkb:matrix.org | We are using a default setup as far as I know. | 15:10 |
| @clarkb:matrix.org | We don't opt out of anything. Even security groups we use but then open up | 15:10 |
| @fungicide:matrix.org | in the past we did observe what looked like stray ra leaking in from devstack running on jobs in our other tenant confusing routing tables | 15:10 |
| @mnasiadka:matrix.org | I'm running `ip monitor route neigh` on the mirror in ca-ymq-1 and it looks like something is getting through | 15:11 |
| @fungicide:matrix.org | the suspicion about announcements originating in devstack running in our other tenant was based on addresses matching devstack's network defaults | 15:15 |
| @clarkb:matrix.org | following up on image upload failures it looks like things are happier now that there is more disk space (and I checked that zl02 cleared out more room after its restart. It did). But all three rax classic uploads are failing on read timeouts now. corvus I know you added read timeouts is it possible the timeout value is simply too aggressive? I think it is 60 seconds | 15:45 |
| @jim:acmegating.com | Clark: give me a clue to find what you see in the logs | 15:46 |
| @clarkb:matrix.org | corvus: `grep -C 100 19b7164c808f49008cd868e5721d2666 /var/log/zuul/launcher-debug.log.1 | less` this is the dfw upload for yesterday's resolute image on zl01. | 15:48 |
| @clarkb:matrix.org | corvus: then look for `TimeoutError: The read operation timed out` | 15:48 |
| @jim:acmegating.com | Clark: that comes from openstacksdk; we set `api_timeout: 60` in clouds.yaml; i don't think that has changed in a long time. that's a glance async task fetch call, which i would expect to be instantaneous. | 15:59 |
| @clarkb:matrix.org | aha | 16:00 |
| @clarkb:matrix.org | and that request is going to the glance tasks api | 16:01 |
| @clarkb:matrix.org | so these failures are happening after we upload to swift and we want to ask glance to import the image from there | 16:01 |
| @clarkb:matrix.org | Dan With: ^ any chance you might know if something changed to make those timeouts happen? | 16:01 |
| @fungicide:matrix.org | that's in classic not flex, right? | 16:04 |
| @clarkb:matrix.org | correct | 16:05 |
| @mordred:waterwanders.com | going WAY out on a limb, but I could imagine rax classic could have accidentally broken the task api, maybe an infrastructure api gateway setting or something. it's always been a bit of an edge-case API. IIRC rax classic is like the only production deployment of that api in existence | 16:09 |
| -@gerrit:opendev.org- Clark Boylan proposed: [opendev/system-config] 1006583: Update etherpad to 3.3.6 https://review.opendev.org/c/opendev/system-config/+/1006583 | 16:14 | |
| -@gerrit:opendev.org- Clark Boylan proposed: [opendev/system-config] 840972: DNM force etherpad failure to hold node https://review.opendev.org/c/opendev/system-config/+/840972 | 16:16 | |
| @mordred:waterwanders.com | infra-root: if you have 3 secs, I'd love to land https://review.opendev.org/c/opendev/system-config/+/1004970 and https://review.opendev.org/c/opendev/system-config/+/996904 - they're both super easy. If people want, I can also push up a followup to 1004970 removing the unused older versions (specifically the ones on bookworm that have no consumers according to codesearch) | 16:22 |
| @jim:acmegating.com | On a different subject -- I noticed a problem with one of the recent zuul performance optimizations. It has caused a small number of jobs to fail with RETRY_LIMIT. I need more debug logging to track down the problem, so I proposed this zuul change: https://review.opendev.org/1006590 -- as soon as that merges, i'll restart the schedulers with it. | 16:36 |
| the problem does not seem widespread, so i think we can keep rolling forward, but if it becomes more of a problem i should be able to come up with a revert. | ||
| @clarkb:matrix.org | corvus: this is a separate retry limit issue to the one I addressed by stopping ze03 then? | 16:38 |
| @jim:acmegating.com | Clark: yes, this shows up with a KeyError exception on the executors | 16:38 |
| @jim:acmegating.com | basically, scheduler thinks it doesn't need to write a secret to the blobstore, but the executor can't find it, so apparently the scheduler was wrong about that but i don't know why | 16:39 |
| @clarkb:matrix.org | ack mostly want to be able to articulate that to someone if they continue to have issues and wonder why when I said rechecks were good to go | 16:39 |
| -@gerrit:opendev.org- Monty Taylor https://matrix.to/#/@mordred:inaugust.com proposed: [opendev/system-config] 1006595: Stop mirring older unused node base images https://review.opendev.org/c/opendev/system-config/+/1006595 | 16:52 | |
| @mordred:waterwanders.com | Clark: ^^ there's the followup cleanup patch | 16:52 |
| -@gerrit:opendev.org- Zuul merged on behalf of Monty Taylor https://matrix.to/#/@mordred:inaugust.com: [opendev/system-config] 996904: Add testing to the mirrored debian tags https://review.opendev.org/c/opendev/system-config/+/996904 | 16:53 | |
| -@gerrit:opendev.org- Zuul merged on behalf of Monty Taylor https://matrix.to/#/@mordred:inaugust.com: [opendev/system-config] 1004970: Start mirroring node:26-trixie https://review.opendev.org/c/opendev/system-config/+/1004970 | 17:11 | |
| @clarkb:matrix.org | held etherpad at 104.130.26.250 lgtm between firefox and chrome (clarkb-test is the name of the pad if you awnt to check too). I'm going to approve 1006583 now. | 17:11 |
| @fungicide:matrix.org | thanks! | 17:13 |
| -@gerrit:opendev.org- Zuul merged on behalf of Monty Taylor https://matrix.to/#/@mordred:inaugust.com: [opendev/system-config] 1006595: Stop mirring older unused node base images https://review.opendev.org/c/opendev/system-config/+/1006595 | 17:17 | |
| @clarkb:matrix.org | For the infra meeting agenda I plan to recap the ze03 disk filling issue and can touch on the other retry limit cause. I'll do an update on image build reliability as well as upload situation. Any other changes/updates to make? | 17:28 |
| -@gerrit:opendev.org- Zuul merged on behalf of Clark Boylan: [opendev/system-config] 1006583: Update etherpad to 3.3.6 https://review.opendev.org/c/opendev/system-config/+/1006583 | 17:52 | |
| @clarkb:matrix.org | I'll monitor the deployment of that upgrade | 17:54 |
| @clarkb:matrix.org | deployment was a success: https://zuul.opendev.org/t/openstack/buildset/d802865a390e44a4858bbb79345f7bbe docker reports the container is healthy. I can reach https://etherpad.opendev.org/p/gerrit-upgrade-3.13 and it renders correctly as far as I can tell | 17:59 |
| @clarkb:matrix.org | corvus: for executors it looks like there may be one more debugging image we can clean up on them to try and free a bit of space on /? | 18:09 |
| @clarkb:matrix.org | I think you mentioned planning to do that as well as restarting ze03 after checking on it? I just want to make sure so that we don't step on each other's toes | 18:10 |
| @jim:acmegating.com | Clark: yep that's still on my list -- i started looking into the disk space but noticed the other problem, i still plan on doing it today | 18:13 |
| @jim:acmegating.com | (i just want to double check some more things about disk space usage before deleting) | 18:13 |
| @clarkb:matrix.org | yup I figured there would be more debugging/inscpection so didn't want to run off and delete things without checking in first. I'll defer to you and let me know if I can help. Though I Have to run errands after lunch | 18:14 |
| @jim:acmegating.com | this is curious: `podman image ls` shows this image on at least several of the executors: | 18:34 |
| `<none> <none> 7bd59683098f 2 days ago 2.21 GB` | ||
| @jim:acmegating.com | it's an executor image, but no repo/tag info | 18:34 |
| @jim:acmegating.com | i'm not sure how that happens or why. timing suggests something with the restart playbook. | 18:35 |
| @jim:acmegating.com | oh that's the currently running image | 18:36 |
| @jim:acmegating.com | i get it.. i'm more used to the docker display where it tells you that | 18:36 |
| @clarkb:matrix.org | we pulled a newer :latest which has confused podman I guess | 18:36 |
| @jim:acmegating.com | i'll accept the confused label this time :) | 18:36 |
| @jim:acmegating.com | it's just podman's way of saying it's an orphaned image | 18:37 |
| @mordred:waterwanders.com | so it's still in the list because there is a container running using it, but it's not showing a tag because the latest tag now points to something else? | 18:38 |
| @jim:acmegating.com | yep | 18:38 |
| @jim:acmegating.com | the docker cli puts a little "U" to the right indicating whether an image is in use; i guess i'm trained on that more than i thought :) | 18:39 |
| @mordred:waterwanders.com | fun how CLI outputs train you isn't it? :) | 18:39 |
| @jim:acmegating.com | yeah, i've run that command a lot :) | 18:39 |
| @jim:acmegating.com | i'm poking around /var/lib/containers/storage/overlay and there's a layer that doesn't belong to any image (at least according to podman image inspect). this is true on ze01 and ze03. but the layer shas are different | 18:46 |
| @jim:acmegating.com | it appears to be part of an executor image | 18:46 |
| @jim:acmegating.com | since ze03 is off, i'm going to delete the last remaining image just to confirm (so there should be no layers) | 18:47 |
| @jim:acmegating.com | okay, on ze03 there are 0 containers, 0 images, 0 volumes. but there are 30 overlayfs layers that still exist, taking up 5GB | 18:49 |
| @jim:acmegating.com | ran `podman system prune -a` -- no change | 18:52 |
| @jim:acmegating.com | there's a sqlite db in the container storage directory; it has 0 entries. | 18:55 |
| @jim:acmegating.com | i think these are well and truly leaked | 18:56 |
| @jim:acmegating.com | `podman system df` confirms no disk usage from images, containers, volumes | 18:58 |
| @jim:acmegating.com | `podman system reset` is what we want to clean that up | 18:59 |
| @jim:acmegating.com | i ran that, and get the 5GB back | 18:59 |
| @jim:acmegating.com | ze03 is now down to 20G used -- add an executor image to that and we're at 22.5G, which is where we were 5 months ago. | 18:59 |
| @jim:acmegating.com | i think we can clean up the intentional executor images on the other executors in order to get a bit of headroom back, but i expect they all have leaked stuff, so we should shut them down and run `podman system reset` on them. | 19:01 |
| @clarkb:matrix.org | Interesting | 19:11 |
| @jim:acmegating.com | i think unbound got borked on ze03 | 19:11 |
| @jim:acmegating.com | yeah, /var/lib/unbound/root.key is 0 bytes | 19:12 |
| @jim:acmegating.com | okay, i just copied that from another host | 19:13 |
| @jim:acmegating.com | seems to be working now, and `docker-compose pull` was successful. i guess the question is whether we trust ze03 is otherwise okay after filling the disk, or if we want to do any more remediation. | 19:14 |
| @fungicide:matrix.org | i think the package maintainer scripts normally create that file | 19:14 |
| @fungicide:matrix.org | but it's not shipped as-is in the unbound package | 19:14 |
| @fungicide:matrix.org | may be a cronjob tried to update it while the rootfs was full | 19:15 |
| @jim:acmegating.com | yeah i think that's what happened (though i think the server itself can rewrite it while running, so that may have been the immediate cause) | 19:15 |
| @fungicide:matrix.org | oh, yeah might have been the unbound daemon | 19:16 |
| @fungicide:matrix.org | regardless, seems likely something tried to replace it but only got as far as the unlink and had no room to make a new one | 19:16 |
| @jim:acmegating.com | i think i'm okay putting ze03 back in service, but i'll wait a bit in case anyone thinks we should do anything else | 19:18 |
| @clarkb:matrix.org | Ya I think restarting unbound may solve it? I'm about to head off on an errand but no concerns from me putting ze03 back into service | 19:21 |
| @jim:acmegating.com | restarting unbound does not solve it, i had to copy the file from another host. but i did that and it is solved now. | 19:22 |
| @clarkb:matrix.org | Ack | 19:28 |
| @fungicide:matrix.org | the `unbound-anchor` package contains a command-line utility that can force update the file, i'm not finding a way to prompt unbound to do it on demand | 19:43 |
| @jim:acmegating.com | yeah that's why i just copied it | 20:03 |
| @jim:acmegating.com | #status log restarted schedulers/web with more blob-store-related debug logging | 20:05 |
| @status:opendev.org | @jim:acmegating.com: finished logging | 20:05 |
| @jim:acmegating.com | #status log ran "podman system reset" on ze03 to clear leaked files and restarted zuul-executor | 20:09 |
| @status:opendev.org | @jim:acmegating.com: finished logging | 20:09 |
| -@gerrit:opendev.org- James E. Blair https://matrix.to/#/@jim:acmegating.com proposed: | 20:17 | |
| - [opendev/system-config] 1006632: Reset podman data on zuul components https://review.opendev.org/c/opendev/system-config/+/1006632 | ||
| - [opendev/system-config] 1006633: Revert "Reset podman data on zuul components" https://review.opendev.org/c/opendev/system-config/+/1006633 | ||
| @clarkb:matrix.org | Anil Belur: thank you for looking at https://review.opendev.org/c/openstack/project-config/+/961499 I left a comment to clarify the bit about the url value. Hope that helps | 21:53 |
| @clarkb:matrix.org | My first pass edit to the meeting agenda is posted. Let me know fi there are other edits otherwise I'll get this sent out shortly | 21:54 |
Generated by irclog2html.py 4.1.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!