| kevko | Good morning | 07:24 |
|---|---|---|
| kevko | I would like to ask what is this https://review.opendev.org/c/openstack/kolla-ansible/+/928619 'Ready to submit' ..it seems arm job is blocking from merge ? Is it something new ? | 07:24 |
| SvenKieske | looks like a gerrit bug to me, weird. | 07:45 |
| tobias-urdin | kevko: perhaps that you have patchset 3 on the parent patch but patchset 4 is merged, a simple rebase and approve again? | 07:50 |
| kevko | tobias-urdin: Shouldn't it be irrelevant? Even if it's in the relation chain, it doesn't matter; the two patch sets don't modify the same lines of code or anything like that. | 07:55 |
| kevko | tobias-urdin: i rebased and will see, thank you for your advice for now | 08:00 |
| opendevreview | Tyler proposed openstack/diskimage-builder master: Fall back to extract-image on ubuntu build https://review.opendev.org/c/openstack/diskimage-builder/+/926748 | 09:31 |
| stephenfin | clarkb: fungi: Would one of you be able to delete the pyparsing-update branch from openstack/cliff. I tried via the web UI but don't see any delete buttons (on https://review.opendev.org/admin/repos/openstack/cliff,branches) | 11:04 |
| *** dhill is now known as Guest3724 | 12:07 | |
| fungi | kevko: what does matter is whether the parent commit id exists on the target branch. if the parent was changed and the child was not rebased onto the new commit, there is no way to merge it directly to the branch (this is a git fundamental even, nothing to do with gerrit) | 12:56 |
| fungi | stephenfin: branch deletion is a permission which can be granted through the repository's acl, similar to branch creation. i can clean up that branch in a few minutes if there are no open changes still targeting it | 12:58 |
| stephenfin | fungi: okay, makes sense. Thanks | 13:20 |
| fungi | #status log Deleted the pyparsing-update branch from openstack/cliff (formerly 993972982739b2db3028278cb4d99be2a713d09c) at Stephen Finucane's request | 14:10 |
| opendevstatus | fungi: finished logging | 14:10 |
| fungi | stephenfin: ^ | 14:10 |
| stephenfin | ty :) | 14:10 |
| fungi | yw | 14:10 |
| frickler | did anyone else see this bunch of mails from our servers with no recipients? | 14:22 |
| fungi | to root/aliases i guess? | 14:22 |
| fungi | oh, ndr messages | 14:23 |
| frickler | "Subject: Mail failure - no recipient addresses" | 14:25 |
| fungi | looks like they started today around 06:02 utc and continued up until 07:08 utc | 14:25 |
| fungi | looking at the one from paste01, this is what was logged by exim: | 14:27 |
| fungi | 2024-09-17 06:02:35 1sqRIJ-009xlk-UX 1sqRIJ-009xlk-UX no recipients found in headers | 14:27 |
| fungi | 2024-09-17 06:02:35 1sqRIJ-009xln-Uz <= <> R=1sqRIJ-009xlk-UX U=Debian-exim P=local S=776 | 14:28 |
| fungi | syslog has a traceback at that time from a process initiated by apt.systemd.daily | 14:32 |
| fungi | ImportError: cannot import name 'HeaderWriteError' from 'email.errors' (/usr/lib/python3.8/email/errors.py) | 14:33 |
| fungi | just after that we have: | 14:33 |
| fungi | Sep 17 06:02:36 paste01 systemd[1]: apt-daily-upgrade.service: Succeeded. | 14:33 |
| fungi | so it looks like something's broken in the e-mail notification part of unattended-upgrade | 14:34 |
| fungi | there were python upgrades which happened in this batch, so maybe a race condition maybe a regression | 14:35 |
| fungi | i guess we should wait and see if it happens again | 14:35 |
| frickler | ah, right, you can see the full traceback in the journal. likely related to the python upgrades I agree | 14:40 |
| johnsom | clarkb I can launch test jobs for those nested virt clouds, but I need to know how to target them. That patch looks like the same labels we normally use.... | 16:16 |
| clarkb | johnsom: yes its the same label. You'll run the jobs and see if they work or not | 16:16 |
| clarkb | adding extra labels is unfortunately a big maintenance burden (its really hard to remove them later) so I want to avoid that if we can | 16:17 |
| johnsom | Ok, so our normal runs "may" land on those hosts. Got it | 16:17 |
| clarkb | yup | 16:17 |
| clarkb | really the only thing we're concerned about is whether the nested virt is particularly unstable since jobs generally work in those regions | 16:18 |
| johnsom | Sure, | 16:18 |
| johnsom | I will keep an eye out and run a few DNM jobs | 16:18 |
| johnsom | clarkb Looking at a patch I posted last night, both openmetal and raxflex instances have worked fine. | 16:24 |
| opendevreview | Merged opendev/zuul-jobs master: Copy DIB elements from project-config https://review.opendev.org/c/opendev/zuul-jobs/+/929140 | 16:24 |
| clarkb | thank you for confirming. Both clouds indicated they expected it to work so good to confirm it | 16:25 |
| opendevreview | Merged opendev/zuul-jobs master: Convert python2 dib element scripts to python3 https://review.opendev.org/c/opendev/zuul-jobs/+/929168 | 16:25 |
| corvus | https://zuul.opendev.org/t/opendev/build/abadb70bd617424098f3d2fe3ac95bea/log/job-output.txt#5860 | 17:17 |
| corvus | interesting -- diskimage build failed on trying to get the zanata archive | 17:17 |
| clarkb | corvus: I don't get a 403 from that now, could just be a momentary failure? | 17:18 |
| corvus | same | 17:18 |
| corvus | maybe that dib element should retry | 17:19 |
| corvus | (i suspect this process may make our build errors more visible :) | 17:19 |
| opendevreview | Clark Boylan proposed zuul/zuul-jobs master: Collection microk8s inspect info in k8s log collection role https://review.opendev.org/c/zuul/zuul-jobs/+/929689 | 17:30 |
| frickler | we likely do not notice most one-off dib failures currently, yes. looking at https://grafana.opendev.org/d/f3089338b3/nodepool3a-dib-status?orgId=1 I also note that a) arm64 builds look failing according to this and b) some old images could get removed, maybe new ones are missing, too? | 17:35 |
| opendevreview | Clark Boylan proposed zuul/zuul-jobs master: Collection microk8s inspect info in k8s log collection role https://review.opendev.org/c/zuul/zuul-jobs/+/929689 | 17:35 |
| clarkb | frickler: yes the dashboard could be udpated to drop old images. re arm64 are they failing again? That is unfortunate. I fI had to guess we're using all the loopback devices again maybe but I haven't looked | 17:37 |
| frickler | clarkb: looks like it: failed to set up loop device: No such file or directory | 17:39 |
| clarkb | ya so for whatever reason we seem to leak those more doing arm builds. One hunch I had is that the arm builds take much much longer so are more likely to be interrpted by udpted nodepool container images that restart the containers. But I don't think we've had a nodepool container update in a while so that may not be it | 17:39 |
| frickler | "Up 2 weeks", so not recently | 17:42 |
| frickler | checking now when the last successful build was and whether we might still have the log from that and the first failure | 17:44 |
| opendevreview | Clark Boylan proposed zuul/zuul-jobs master: Bump the default ensure-kubernetes microk8s version to 1.31/stable https://review.opendev.org/c/zuul/zuul-jobs/+/929689 | 17:45 |
| frickler | centos-9-stream-arm64-fe857118d17141abaed43b10ed7fbd64 ready 03:11:53:43 - that seems to be the most recent one | 17:46 |
| clarkb | but the leaks can happen slowly since the last boot so it might be one a day or whatever | 17:46 |
| clarkb | but having the container running for 2 weeks implies it isn't container restarts that are the primary cause | 17:47 |
| frickler | looks like openeuler failed after the c9s build with a different issue. and the rocky9 build after that then shows the same loop device failure as above | 17:49 |
| frickler | so my suggestion would be to pause openeuler arm64 and clean up the server and try again, but I won't have time to do that myself this week | 17:50 |
| clarkb | ya the loopback is only used when compiling the final image together from the on disk stuff | 17:50 |
| clarkb | if the on disk stuff fails early we don't hit the loopback issue | 17:50 |
| clarkb | and ya maybe openeuler is failing in such a way that we leak loopback setup or something | 17:51 |
| opendevreview | Clark Boylan proposed zuul/zuul-jobs master: Bump the default ensure-kubernetes microk8s version to 1.31/stable https://review.opendev.org/c/zuul/zuul-jobs/+/929689 | 17:56 |
| frickler | the second to last line in https://nb04.opendev.org/openEuler-22-03-LTS-arm64-5de13663a7fa45ebaf1df171d62ad94c.log is "LOOPDEV=/dev/loop7", so I was assuming that to happen, yes. I haven't been able to see what the actual failure is, though, maybe just a timeout? | 18:00 |
| opendevreview | Merged opendev/zuul-jobs master: Add debian-bullseye image https://review.opendev.org/c/opendev/zuul-jobs/+/929141 | 18:01 |
| opendevreview | Clark Boylan proposed zuul/zuul-jobs master: Bump the default ensure-kubernetes microk8s version to 1.31/stable https://review.opendev.org/c/zuul/zuul-jobs/+/929689 | 18:15 |
| opendevreview | Clark Boylan proposed zuul/zuul-jobs master: Bump the default ensure-kubernetes microk8s version to 1.31/stable https://review.opendev.org/c/zuul/zuul-jobs/+/929689 | 18:27 |
| fungi | NeilHanlon: https://opendev.org/opendev/system-config/src/branch/master/playbooks/roles/mirror-update/files/ in case you are looking for script inspiration | 19:55 |
| NeilHanlon | thanks! | 20:00 |
| opendevreview | Clark Boylan proposed zuul/zuul-jobs master: Bump the default ensure-kubernetes microk8s version to 1.31/stable https://review.opendev.org/c/zuul/zuul-jobs/+/929689 | 21:50 |
| opendevreview | Clark Boylan proposed zuul/zuul-jobs master: Bump the default ensure-kubernetes microk8s version to 1.31/stable https://review.opendev.org/c/zuul/zuul-jobs/+/929689 | 22:05 |
| opendevreview | Merged openstack/diskimage-builder master: Adapt to upstream CentOS Stream mirror changes https://review.opendev.org/c/openstack/diskimage-builder/+/922352 | 23:02 |
| opendevreview | Merged openstack/diskimage-builder master: dib-functests: run on bookworm https://review.opendev.org/c/openstack/diskimage-builder/+/922697 | 23:13 |
| opendevreview | Merged openstack/diskimage-builder master: Drop logic for old python versions https://review.opendev.org/c/openstack/diskimage-builder/+/927781 | 23:59 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!