| opendevreview | OpenStack Proposal Bot proposed openstack/project-config master: Normalize projects.yaml https://review.opendev.org/c/openstack/project-config/+/976568 | 02:35 |
|---|---|---|
| opendevreview | Merged openstack/project-config master: Normalize projects.yaml https://review.opendev.org/c/openstack/project-config/+/976568 | 07:17 |
| stephenfin | fungi: You probably know this already, but it seems the scrapers have killed opendev.org again :( I'm getting timeouts loading the web UI and cloning via opendev (`Connection timed out after 300013 milliseconds`) | 10:57 |
| stephenfin | I'm far from the first to say it but these scrapers are going to kill the open web | 10:57 |
| stephenfin | what's left of it that is | 10:57 |
| fungi | stephenfin: looking, thanks | 11:46 |
| fungi | stephenfin: are you still seeing it happen? maybe we can figure out which backend is hitting timeouts for you, because it's snappy for me on whichever one i'm being load-balanced to | 11:47 |
| stephenfin | fungi: it is, but not for sean-k-mooney either. Let me know what you'd like to check | 11:49 |
| sean-k-mooney | stephenfin: what ip is it resolvign too for a start for you | 11:50 |
| fungi | stephenfin: sean-k-mooney: the subject on the ssl cert for the https://opendev.org/ i'm looking at says CN=gitea11.opendev.org | 11:50 |
| fungi | there is only one ip address, it's not round-robin dns | 11:50 |
| fungi | just an haproxy server directing clients to one of six gitea servers based on a source address hash | 11:51 |
| sean-k-mooney | fungi: well im defualting to ipv6 we are on the same network but im not sure if stephen is hiting the ipv4 endpoint or not | 11:51 |
| stephenfin | dig +short opendev.org -> 38.108.68.97 | 11:51 |
| fungi | yeah, i'm connecting via ipv6 to the haproxy address as well | 11:52 |
| sean-k-mooney | which is 2604:e100:3:0:f816:3eff:fe5a:c9d7 | 11:52 |
| sean-k-mooney | https://paste.opendev.org/show/bEQGkBkPUpnDyZz3bAMs/ | 11:53 |
| stephenfin | looks like I'm hitting gitea09.opendev.org | 11:53 |
| stephenfin | https://paste.opendev.org/show/bRTSc3vL5dWTbheZoYHu/ | 11:53 |
| fungi | gitea09 does seem to have about twice the load average as the other backends, though still not exceptionally high | 11:54 |
| sean-k-mooney | 13 for me https://paste.opendev.org/show/b2gXrULhcG2z2rqLDPjf/ | 11:55 |
| sean-k-mooney | i asume that is base don my source ipv6 address | 11:55 |
| sean-k-mooney | stephenfin: you should be able to change your ipv6 address to privay mode or otherwise rothate it | 11:56 |
| fungi | apache on gitea09 also has a bunch of open/waiting worker slots, so it doesn't look overloaded with concurrent requests | 11:56 |
| fungi | are you seeing any packet loss trying to ping 38.108.68.97 or 2604:e100:3:0:f816:3eff:fe5a:c9d7 maybe? | 11:57 |
| sean-k-mooney | no icmp frorm the v6 address | 11:58 |
| stephenfin | pings for both look fine. 150-250ms latency but not dropped packets | 11:58 |
| sean-k-mooney | all lost but the v4 adress pings fine for me | 11:58 |
| stephenfin | oh, the v6 one wfm | 11:58 |
| sean-k-mooney | weird | 11:58 |
| sean-k-mooney | the 150-200 latecy is normal since i think thik tis is on the east or maybe even west cost of the us | 12:00 |
| sean-k-mooney | best case to new york would be about 80 ms | 12:01 |
| sean-k-mooney | so 150-200 woudl be similar to aws us-west-2 | 12:03 |
| fungi | yes, the gitea servers and accompanying load balancer are in a vexxhost region in san jose, california, usa so it's on the west coast of north america | 12:03 |
| fungi | when you get the "Connection timed out after..." error, does it appear to be an error message the gitea frontend is giving about timing out on a database query, or an error message your browser is generating? | 12:06 |
| stephenfin | the connection timed out error is comming from `git fetch` | 12:07 |
| stephenfin | but the website is also timing out via Firefox and curl | 12:07 |
| fungi | even the top-level https://opendev.org/ url? | 12:09 |
| stephenfin | yup :( | 12:10 |
| stephenfin | review.opendev.org is fine | 12:10 |
| fungi | i think that one doesn't trugger any background database queries, so we can probably rule out database overload | 12:10 |
| stephenfin | as are various other sites like docs.openstack.org (different hosts, naturally) | 12:11 |
| fungi | stephenfin: review.opendev.org is in a different vexxhost region, quebec canada | 12:11 |
| fungi | east side of of north america | 12:11 |
| stephenfin | let me try disconnecting from the VPN to see if it's something funky internally. will lose IRC access (my bouncers behind the firewall) | 12:11 |
| * stephenfin brb | 12:11 | |
| fungi | and docs.openstack.org is in a rackspace legacy region in dallas, texas (~middle of north america) | 12:12 |
| fungi | https://mirror.sjc1.vexxhost.opendev.org/ is the other web-reachable service we have in vexxhost's san jose region, so maybe that one's acting better/worse from your network? | 12:14 |
| stephenfin | yeah, no dice. disconnecting from the firewall didn't help | 12:14 |
| stephenfin | https://mirror.sjc1.vexxhost.opendev.org/ loads just fine | 12:14 |
| fungi | that's useful data at least | 12:15 |
| fungi | so the two of you are seeing slow performance from the gitea09 and gitea13 backends... | 12:15 |
| fungi | no packet loss reaching the load balancer | 12:16 |
| fungi | and you're getting the same behavior whether you git fetch/curl over ipv4 or ipv6? | 12:16 |
| stephenfin | oh, interesting: I can fetch it if I curl over ipv4 | 12:19 |
| stephenfin | i.e. 'curl https://opendev.org/ -4' | 12:19 |
| stephenfin | but not ipv6 | 12:19 |
| fungi | i also see if i switch between ipv6 and ipv4 i get different backends (gitea11 vs gitea14) | 12:19 |
| fungi | though i'm still getting immediate response/content from both | 12:20 |
| stephenfin | if I curl with ipv6, I get routed to 2604:e100:3:0:f816:3eff:fe5a:c9d7 (which hangs) | 12:22 |
| stephenfin | ipv4 resolves to 38.108.68.97 which is fine | 12:22 |
| fungi | yeah, those are the correct ip addresses for the load balancer at least | 12:22 |
| stephenfin | maybe it's something local, but sean-k-mooney and I are on the same network and this worked fine up until early this week, so idk what could have changed | 12:22 |
| sean-k-mooney | i was checking my unify dashbaord but im not seeing any blocks or issue not that i really know what to look at in this case | 12:23 |
| frickler | stephenfin: can you try traceroute6? | 12:24 |
| fungi | well, ipv4 and ipv6 can end up following different routes through the internet depending on bgp announcements and filters in various providers along the way, though it doesn't sound like out and out packet loss or else you'd see it impact icmp and not just tcp | 12:24 |
| frickler | fwiw, looks like vexxhost still hasn't managed to publish proper route objects for their /48s, getting immediate "no route to host" within AS3320 | 12:24 |
| fungi | yeah, so presumably even ping (icmp echo) to that address is timing out for you, at least | 12:25 |
| sean-k-mooney | https://termbin.com/yn44 that mine | 12:26 |
| stephenfin | frickler: no luck with either ipv4 or ipv6, weirdly https://paste.opendev.org/show/bBhdHExVnvLhUFlwSkW1/ | 12:27 |
| sean-k-mooney | stephenfin: im tempted to say turn you wifi off an don agao or other wise rotate your ipv6 adress in network manager | 12:27 |
| stephenfin | yeah, might just reboot 😅 It's looking increasingly like a "me" issue | 12:27 |
| sean-k-mooney | in network manger in the ipv6 section uner privacy there is a option to select private or public ipv6 adress but also might be good to confirm what namesever your using and if its the same as mine | 12:28 |
| sean-k-mooney | the network shoudl be handing out the same info to both of us but maybe your laptop is out of sync | 12:29 |
| fungi | also weird that this problem seems to come and go... is it usually at a particular time of day that it starts or clears up, can you tell? | 12:29 |
| opendevreview | Stephen Finucane proposed openstack/project-config master: Onboard more xstatic repos (1/2) https://review.opendev.org/c/openstack/project-config/+/976612 | 12:29 |
| opendevreview | Stephen Finucane proposed openstack/project-config master: Sync xstatic package to gerritbot https://review.opendev.org/c/openstack/project-config/+/976613 | 12:29 |
| opendevreview | Stephen Finucane proposed openstack/project-config master: Onboard more xstatic repos (2/2) https://review.opendev.org/c/openstack/project-config/+/976614 | 12:29 |
| stephenfin | fungi: can't say. iirc my issue yesterday was just with pulling the upper-constraint.txt file and I had no issue cloning 30+ xstatic repos | 12:40 |
| * stephenfin reboots to see if that helps | 12:40 | |
| stephenfin | Windows-style troubleshooting ftw 🙈. All appears to be well, and I can once again curl via ipv4 or ipv6 | 12:44 |
| fungi | huh, weird? | 12:47 |
| fungi | wouldn't expect just a reboot to affect things between your network and the gitea servers, so indeed might have been a more localized temporary issue | 12:48 |
| sean-k-mooney | whats odd is before they rebooted tehy dissbled and renebaled ipv6 and ipv4 seperatly | 12:55 |
| sean-k-mooney | which didnt fix it | 12:55 |
| fungi | stale/extra v6 default route? | 12:56 |
| fungi | i've seen problems before where stray route advertisements with long timeouts get picked up by the client and then packets are sent to a dead/nonexistent gateway | 12:57 |
| fungi | maybe next time look at `ip -6 route show` and see if it's correct or changes after a reboot | 12:58 |
| opendevreview | Stephen Finucane proposed openstack/project-config master: Onboard more xstatic repos (1/2) https://review.opendev.org/c/openstack/project-config/+/976612 | 13:01 |
| opendevreview | Stephen Finucane proposed openstack/project-config master: Onboard more xstatic repos (1/2) https://review.opendev.org/c/openstack/project-config/+/976612 | 13:02 |
| opendevreview | Stephen Finucane proposed openstack/project-config master: Sync xstatic package to gerritbot https://review.opendev.org/c/openstack/project-config/+/976613 | 13:02 |
| opendevreview | Stephen Finucane proposed openstack/project-config master: Onboard more xstatic repos (2/2) https://review.opendev.org/c/openstack/project-config/+/976614 | 13:02 |
| stephenfin | fungi: Ack, will do. Though hopefully that's the end of it | 13:10 |
| stephenfin | On a totally unrelated note, those patches ^ could do with a once over. I believe those two packages are the sole remaining xstatic dependencies that don't have a Git repo attached to them. There are three others but they're hosted on github.com/xstatic-py and I'm working with the maintainers there to bump those. I've already bumped https://pypi.org/project/XStatic-term.js/ | 13:11 |
| stephenfin | clarkb also (when you're about) ^ | 13:11 |
| fungi | stephenfin: you've got it, they're queued up in my client for as soon as i'm unburied from other morning tasks | 13:13 |
| stephenfin | ty | 13:13 |
| fungi | stephenfin: as a point of process, we should be getting tmazur's +1 on these as ptl, or on the new patchset of the associated governance change, before approving | 13:21 |
| stephenfin | Understood. I'm not sure what timezone they're in but based on yesterday they should be around later this afternoon | 13:21 |
| stephenfin | we can wait a few hours to +W | 13:22 |
| fungi | yeah, i'm also not entirely opposed to expediting these and assuming they'll be okay given the circumstance, should it become necessary | 13:23 |
| stephenfin | If that's the case, I think rdopiera's ack, along with tmazur's ack of https://review.opendev.org/c/openstack/project-config/+/976496 and bulk approvals of updates for the existing xstatic packages should be sufficient | 13:25 |
| stephenfin | I would really like to get this closed out this week if possible | 13:26 |
| opendevreview | Elod Illes proposed openstack/openstack-zuul-jobs master: Prepare job templates for stable/2026.1 testing runtime https://review.opendev.org/c/openstack/openstack-zuul-jobs/+/976619 | 13:44 |
| opendevreview | Elod Illes proposed openstack/openstack-zuul-jobs master: Add stable/2026.1 to periodic-stable templates https://review.opendev.org/c/openstack/openstack-zuul-jobs/+/976624 | 13:53 |
| opendevreview | Takashi Kajinami proposed openstack/project-config master: Onboard more xstatic repos (1/2) https://review.opendev.org/c/openstack/project-config/+/976612 | 13:54 |
| opendevreview | Stephen Finucane proposed openstack/project-config master: Sync xstatic package to gerritbot https://review.opendev.org/c/openstack/project-config/+/976613 | 13:55 |
| opendevreview | Stephen Finucane proposed openstack/project-config master: Onboard more xstatic repos (2/2) https://review.opendev.org/c/openstack/project-config/+/976614 | 13:55 |
| opendevreview | Takashi Kajinami proposed openstack/project-config master: Onboard more xstatic repos (2/2) https://review.opendev.org/c/openstack/project-config/+/976614 | 13:55 |
| *** vhari_ is now known as vhari | 15:00 | |
| clarkb | stephenfin: fungi: looks like there is also an invite for the openstackci pypi account to join the xstatic-term.js project on pypi | 15:57 |
| fungi | i can take care of that in a moment unless you already have | 15:57 |
| clarkb | I have not. Would be great if you can | 15:57 |
| fungi | on it | 16:00 |
| stephenfin | clarkb: fungi: more context here https://github.com/takluyver/XStatic-termjs/pull/3 | 16:53 |
| fungi | thanks! | 16:54 |
| stephenfin | as noted in the last comment, I don't believe we want to migrate this to opendev.org, but at least this leaves the door open in the future | 16:54 |
| fungi | stephenfin: clarkb: https://pypi.org/project/XStatic-term.js/ now shows openstackci in the maintainers list | 17:08 |
| opendevreview | Merged openstack/project-config master: Onboard more xstatic repos (1/2) https://review.opendev.org/c/openstack/project-config/+/976612 | 17:44 |
| opendevreview | Merged openstack/project-config master: Sync xstatic package to gerritbot https://review.opendev.org/c/openstack/project-config/+/976613 | 17:44 |
| opendevreview | Merged openstack/project-config master: Onboard more xstatic repos (2/2) https://review.opendev.org/c/openstack/project-config/+/976614 | 18:13 |
Generated by irclog2html.py 4.0.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!