| corvus | https://zuul.opendev.org/t/opendev/build/3a0987f20e5142e4abd6bd29b78d0441/console#4/0/28/ubuntu-noble | 00:03 |
|---|---|---|
| corvus | so far so good re no_log | 00:03 |
| corvus | now we can update that to return some more useful info | 00:04 |
| corvus | we could give that task a timeout so that it can fail internally and return more data too | 00:04 |
| clarkb | we're still getting flaky package installs from upstream mirrors on the system-config-run jobs | 00:41 |
| clarkb | we don't use our own mirrors for those because they arem eant to look like prod | 00:41 |
| clarkb | I'll just have to followup tomorrow morning | 00:41 |
| opendevreview | Vladimir Kozhukalov proposed openstack/project-config master: Retire openstack-helm-infra repo https://review.opendev.org/c/openstack/project-config/+/951237 | 06:06 |
| frickler | mnaser: fwiw from AS3320 I'm now getting "no route to host" for opendev.org (2604:e100:3::/48), while review.opendev.org (2604:e100:1::/48) is working fine. I'd also very much like to see some postmortem report on these repeated networking issues | 07:04 |
| frickler | I'm also having partial connectivity issues via IPv4 to review.o.o for some weeks, where only some of the flows are affected, looking like there is a single broken patch within a loadbalancing bundle within cogent (AS174), but they keep saying "all the pings work fine for us" :-( | 07:25 |
| frickler | ah, wrong, the latter affects all of 38.108.68.0/24 for me, so including gitea (opendev.org), not gerrit | 07:31 |
| mnaser | frickler: we cant really do much traffic steering outside our network. since cogent is a vendor of ours, if you can show that it's clearly broken, i can open a ticket, but i cannot guarantee anything | 11:56 |
| mnaser | at the end of the day, it's _really_ hard for us to solve issues that are happening on networks that don't belong to us, opendev.org was not even affected by yesterday, that was another site | 11:57 |
| *** amoralej_ is now known as amoralej | 12:09 | |
| fungi | been right there myself more times than i can count, i feel for you, hope cogent will get their act together but i know backbone providers too well to expect that | 13:11 |
| fungi | unrelated, i need to run some errands on another nearby island today, so will probably not be around the keyboard between 14:30 and 17:00 utc, in case anyone's looking for me | 13:12 |
| opendevreview | Merged openstack/project-config master: Retire openstack-helm-infra repo https://review.opendev.org/c/openstack/project-config/+/951237 | 14:10 |
| fungi | looks like our system-config jobs are still choking on upstream ubuntu mirror server timeouts | 14:23 |
| fungi | okay, heading out, back in ~2.5 hours hopefully | 14:33 |
| clarkb | fungi: ack. Thanks for rechecking those changes. Though it looks like the us ubuntu mirror pool is still struggling today | 14:42 |
| clarkb | we already haev failures hitting 2620:2d:4002:1::103 which iirc is the same location we had problems with yesterday | 14:42 |
| clarkb | oh though I think it is saying a specific backend is unreachable? | 14:44 |
| clarkb | https://status.canonical.com/ indicates all is well. If we hit failures again I wonder if we should maybe ask canonical/ubuntu about it | 14:46 |
| opendevreview | Merged opendev/system-config master: Switch from netcat to netcat-openbsd package https://review.opendev.org/c/opendev/system-config/+/951178 | 16:13 |
| opendevreview | Merged opendev/system-config master: Install python3-venv for venv creation on mirror-update https://review.opendev.org/c/opendev/system-config/+/951214 | 16:13 |
| JayF | FWIW, there are reports in #gentoo-infra of outages in ubuntu archives as well | 16:18 |
| clarkb | JayF: the two changes aboev are the two that I've been fighting to get in that hit that and they made it through check and gate finally so maybe things are better? but good to know its observable elsewhere | 16:21 |
| JayF | yeah, I did the reciprocal as well. Just nice for everyone to have confirmation they aren't seeing a problem :D | 16:22 |
| JayF | **the only ones seeing a problem | 16:23 |
| clarkb | infra-root I decided to move /usr/local/afsmonvenv aside on mirror-update02 since it was half created and I'm not sure how ansible's pip module will handle that | 16:25 |
| clarkb | and the deploy failed in bootstrap bridge due to an apt update problem | 16:27 |
| clarkb | so maybe ubuntu isn't happy again | 16:27 |
| clarkb | I'll figure out reenqueing it | 16:27 |
| clarkb | `zuul-client enqueue --tenant openstack --pipeline deploy --project opendev.org/opendev/system-config --change 951214,1` and it is reenqueed | 16:29 |
| clarkb | "msg": "Failed to update apt cache: unknown reason" | 16:32 |
| clarkb | this happened again. I'll try to manually do an apt-get update | 16:32 |
| clarkb | Could not connect to security.ubuntu.com:80 (2620:2d:4000:1::103). - connect (111: Connection refused) | 16:33 |
| clarkb | so this does indeed appear to be the same issue. I'll wait a bit before trying again | 16:33 |
| clarkb | but given those issues I think trying to build entirely new servers is also a bit of a lost cause (as it will randomly fail) | 16:34 |
| clarkb | I should probably focus on something else until this is happier | 16:34 |
| JayF | maybe they are behind cogent too :| | 16:34 |
| ykarel | Hi can someone create autohold for job neutron-functional-1, change 950303, need to debug a random issue | 16:47 |
| clarkb | ykarel: yes one moment | 16:49 |
| clarkb | ok that should be in place | 16:50 |
| ykarel | clarkb, thx very much | 16:51 |
| clarkb | I've reeneuqued the afsmonvenv change deployment because bridge apt-get update succeeded for me just now | 16:55 |
| clarkb | success! | 17:00 |
| clarkb | but ya booting new nodes is likely to be flaky while the mirrors are sad so I'll pause here ready to deploy a new mirror-update and new zookeeper and wait on that for when we think mirrors are happy again | 17:01 |
| fungi | good to see those merged and deployed successfully at least | 17:16 |
| clarkb | https://grafana.opendev.org/d/9871b26303/afs?orgId=1&from=now-6h&to=now&timezone=utc I think this shows we're continuing to report afs monitoring stats to graphite/grafana | 17:51 |
| clarkb | (it runs every half hour) | 17:51 |
| clarkb | and you can see a slight change in some values at 17:30 UTC | 17:51 |
| clarkb | ykarel: that jobs completed and then ode is held. Do you have an ssh key posted somewhere I can put on the host? | 17:54 |
| clarkb | ykarel: or feel free to PM me a pubkey | 17:54 |
| ykarel | clarkb, i injected already with add authorized keys role | 17:54 |
| clarkb | ah | 17:54 |
| nhicher[m] | clarkb fungi: FYI, I created a bug report for openafs on centos-9 and I've found today that the issue is dkms now require 'yes' or 'no' values (it's true on dkms.conf provided by openafs) https://github.com/dell/dkms/blob/main/dkms.in#L824. You should not be impacted for NO_WEAK_MODULES is only used on rhel/opensuse, on others distributions is ignored according to the man page of dkms | 20:06 |
| fungi | nhicher[m]: thanks for the follow-up! i suppose that's good news for us, but it prompted us to go ahead and update our openafs package build anyway which isn't a total waste ;) | 20:17 |
| clarkb | nhicher[m]: I think I'd be happy to accept a patch to our package build script that does s/NO_WEAK_MODULES=true/NO_WEAK_MODULES=yes/ if you can figure out the paths and where in the process to apply that | 20:39 |
| clarkb | that is a minimal update that we can cleanup when upstream has fixed it | 20:39 |
| fungi | agreed, sounds fine to me | 20:40 |
| nhicher[m] | clarkb: sure, I will have a look, thanks =) | 20:40 |
| opendevreview | Clark Boylan proposed opendev/engagement master: Fix data accuracy bugs https://review.opendev.org/c/opendev/engagement/+/950471 | 21:16 |
| opendevreview | Merged opendev/engagement master: Collect reviewer and maintainer counts per project https://review.opendev.org/c/opendev/engagement/+/950369 | 22:22 |
| opendevreview | Merged opendev/engagement master: Properly query for opened changes during date range https://review.opendev.org/c/opendev/engagement/+/950442 | 22:23 |
| opendevreview | Merged opendev/engagement master: Fix data accuracy bugs https://review.opendev.org/c/opendev/engagement/+/950471 | 22:25 |
Generated by irclog2html.py 4.0.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!