| opendevreview | Michal Nasiadka proposed openstack/project-config master: Switch pcu job to target lint-requirements.txt for kolla https://review.opendev.org/c/openstack/project-config/+/986482 | 06:06 |
|---|---|---|
| opendevreview | Merged openstack/project-config master: Switch pcu job to target lint-requirements.txt for kolla https://review.opendev.org/c/openstack/project-config/+/986482 | 07:21 |
| *** ykarel_ is now known as ykarel | 07:44 | |
| opendevreview | Bartosz Bezak proposed openstack/project-config master: kayobe: Add reviewer tier with +2 rights https://review.opendev.org/c/openstack/project-config/+/986493 | 07:57 |
| opendevreview | Merged openstack/project-config master: watcher: Add devstack-plugin-prometheus https://review.opendev.org/c/openstack/project-config/+/986386 | 14:42 |
| dtantsur | Folks, is it only me or https://docs.openstack.org/releasenotes/ironic/ keeps arbitrary returning HTTP 403? | 15:49 |
| frickler | dtantsur: not for me, so likely just for you and some of our AI plague that share the same user agent. are you using some weird or older browser maybe? | 16:05 |
| dtantsur | frickler: nope, the default Firefox on Fedora | 16:07 |
| dtantsur | I also don't recall having issues on Anubis-enabled sites | 16:08 |
| JayF | As a fellow Linux user, I have seen more than once my rare-ish user agent (Chrome or Firefox on Gentoo Linux) get banned. | 16:08 |
| JayF | or at least slowed | 16:08 |
| JayF | Is it possible you're also using AI agents that might be hitting opendev properties and getting frozen out? e.g. claude fetching stuff? | 16:08 |
| dtantsur | Not right now. And I mostly use Claude for development only, it rarely access anything. | 16:10 |
| dtantsur | I wonder if someone with access to the logs could grep for 403 and releasenotes/ironic to see why it happened | 16:11 |
| frickler | hmm, it looks like maybe we are seeing AFS timeouts, sadly other admins are travelling, so not sure when someone will be able to look closer | 16:22 |
| fungi | checking in on a break, static03 where docs.openstack.org is hosted complained in dmesg about waiting for a busy volume at 10:36:38 utc (over 6 hours ago) but no warnings since | 16:47 |
| fungi | i don't see any 403 responses for ironic release notes requests where the user agent claimed to be firefox | 16:50 |
| fungi | what's the exact url you're requesting? | 16:50 |
| fungi | dtantsur: knowing your exact user agent string would help too, but elsewhere you mentioned that going to the parent releasenotes page first was allowing you to the browse the ironic releasenotes page from there... is there maybe some sort of proxy between your machine and the site? | 17:11 |
| dtantsur | fungi: not that I'm aware of. But I'm on RH VPN currently, which can affect anything too. | 17:12 |
| fungi | yeah, i wonder if there's a transparent proxy that has cached a 403 for the direct url but then invalidates that when you browse another page referring to it or something | 17:12 |
| fungi | that's a very weird symptom and not something i'd expect. also i wouldn't expect any afs timeouts or apache overload to return a 403 permission denied error | 17:13 |
| fungi | and there's no anubis in front of those sites either | 17:14 |
| clarkb | ya to be clear we did have issues with some over zealous waf and mod rewrite rules during the period of time where things were really bad nad we were trying to get back under control. Once we got back under control we reverted back to the old state prior to that | 17:18 |
| clarkb | so there shouldn't be any new rules blocking this | 17:18 |
| clarkb | My suggestion would be to make a request to a specific url now so that we can look at that specific request more easily knowing it is you and the nwork from there | 17:18 |
| dtantsur | the problem is not persistent, it comes and goes | 17:21 |
| dtantsur | let me hit https://docs.openstack.org/releasenotes/ironic/ again | 17:21 |
| dtantsur | just did | 17:21 |
| fungi | looks like you got a 200/ok response that time? | 17:22 |
| fungi | i see it returned in the access log | 17:23 |
| frickler | looks like every mention of an URL here triggers a couple of requests from thelounge bots, too ;) | 17:24 |
| fungi | dtantsur: looks like that came in from a residential isp in germany, that's you? | 17:25 |
| opendevreview | Merged openstack/project-config master: Retire requestsexceptions https://review.opendev.org/c/openstack/project-config/+/979808 | 17:25 |
| dtantsur | fungi: sounds like me | 17:27 |
| dtantsur | esp. if it geolocates to Munich | 17:27 |
| clarkb | yes making that request was helpful | 17:28 |
| clarkb | at ~15:30 UTC (~2 hours ago) there are 403 responses to that page with your ip address listed as the requestor. THen the apache error log complains about being unable to get the file in afs | 17:29 |
| fungi | dtantsur: okay, after more digging, it looks like there could be weird conditions where afs times out and confuses apache into returning a 403 permission denial (maybe related to file permissions on the parent directory) | 17:29 |
| clarkb | I think what is unexpected here is that apache would return a 403 error when afs fails (I would've expected a 404 or 500 instead) | 17:29 |
| clarkb | but it seems related to that whcih is probably related to the server being overwhelmed earlier which frickler addressed | 17:29 |
| dtantsur | okay, good to know, thanks for looking into it! | 17:29 |
| fungi | theory time: requesting the parent directory gets it back into the afs client cache, then apache no longer thinks there's a permissions problem | 17:30 |
| fungi | and then eventually that's evicted from the cache and the error returns? | 17:30 |
Generated by irclog2html.py 4.1.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!