| priteau | Hello. Is there a known issue with docs.openstack.org? Pages are slow or unable to load. | 08:38 |
|---|---|---|
| bbezak | and releases.openstack.org is not reachablge | 10:29 |
| bbezak | *reachable | 10:29 |
| bbezak | or if working it is very slow | 10:29 |
| fungi | docs.openstack.org and releases.openstack.org are on the same server | 11:16 |
| fungi | i agree it's basically unresponsive. i'll have to look at it in a bit once i get to my workstation | 11:17 |
| fungi | dmesg on static.o.o is complaining semi-regularly about "afs: Warning: We are having trouble keeping the AFS stat cache trimmed down under the configured limit" | 12:31 |
| fungi | afs servers seem generally healthy at first glance | 12:33 |
| fungi | of course, content is being served quickly now, so whatever it was seems to have subsided in the past hour. i'll check graphs | 12:34 |
| slittle | Is review.opendev.org down? | 12:37 |
| fungi | it's rendering for me | 12:38 |
| fungi | last reboot was a little over 6 months ago, system load is nominal | 12:38 |
| fungi | the current gerrit service process has been running there since about 2 weeks | 12:39 |
| fungi | no, sorry, 6 weeks | 12:39 |
| fungi | i can't remember what month it is, moar coffee | 12:39 |
| slittle | I think it was a temporary routing issue with my isp/vpn | 12:46 |
| slittle | it's working now | 12:46 |
| *** dhill is now known as Guest5098 | 12:46 | |
| jaltman | @fungi: "afs: Warning: We are having trouble keeping the AFS stat cache trimmed down under the configured limit" is logged at most once every four hours. It means that the maximum number of vnode cache (aka vcache) entries have been used or exceeded. | 12:48 |
| jaltman | The next line in dmesg will be afs: "If AFS access seems slow, consider raising the -stat setting for afsd." | 12:48 |
| jaltman | The warning is unrelated to the afs servers. | 12:49 |
| fungi | jaltman: yes, thanks. that's why i mentioned it, i suspect the client (a webserver) may be getting overwhelmed by another blast of llm training crawlers | 12:50 |
| fungi | and we don't have the cache tuned well for that sort of activity | 12:51 |
| jaltman | if crawlers are permitted to traverse /afs then there isn't much tuning that can help because there is no working set that can fit within the cache | 12:53 |
| jaltman | in my opinion the warning message is a bit silly. its generated when the vcache entry free list is empty but that by itself is not an indication that the cache is stressed. LRU recycling is perfectly normal behavior unless the cache is overly large. | 12:55 |
| fungi | makes sense, thanks! | 12:56 |
| fungi | and yes, i too don't think that there's much cache tuning can do in the face of crawlers, we'll just keep playing whack-a-mole with various fingerprints on the webserver side to keep their requests from resulting in fetches from afs | 12:58 |
| SvenKieske | fungi: is the issue(?) resolved? we had problems accessing https://tarballs.opendev.org/ | 13:24 |
| fungi | SvenKieske: same server as docs and releases, yes. it seems to be behaving fine at the moment, so whatever happened was temporary and had already ceased when i started looking into it | 13:26 |
| fungi | there's a good chance it was overloaded by a flood of requests, but i'd need to do some apache log analysis to confirm that theory | 13:27 |
| fungi | SvenKieske: do you happen to have approximate times you saw issues with the tarballs site? | 13:36 |
| clarkb | https://static.opendev.org/ and https://static.opendev.org/project/ give a possibly incomplete listing of stuff hosted by AFS and the static server if anyone is wondering | 14:58 |
| SvenKieske | fungi: sorry for the desync'ed answer (funny I just wait for a galera cluster sync): it was around 08:41 UTC my colleague told me | 15:00 |
| *** dhill is now known as Guest5120 | 16:34 | |
| opendevreview | Julia Kreger proposed openstack/diskimage-builder master: Reduce LVM extent usage https://review.opendev.org/c/openstack/diskimage-builder/+/930950 | 17:36 |
| opendevreview | Merged openstack/project-config master: Temporarily remove release docs semaphores https://review.opendev.org/c/openstack/project-config/+/930709 | 19:21 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!