| ykarel | Hi releases.openstack.org is down, is that known issue? | 04:11 |
|---|---|---|
| ykarel | most of the jobs red due to this, example failure build https://zuul.opendev.org/t/openstack/build/88e88f89dddb41888430f6ad7be76739 | 04:13 |
| ykarel | also sent over openstack-discuss to hold recheck https://lists.openstack.org/archives/list/openstack-discuss@lists.openstack.org/thread/X4YBRF6LOPJTPHRRLRYOQ6W7Y5RWKYNI/ | 05:03 |
| ykarel | same is with docs.openstack.org, also down | 06:15 |
| frickler | ykarel: yes, there was an issue again like on Friday, but mnasiadka already restarted it | 08:58 |
| tkajinam | sent out a reply to broadcast that :-) | 09:40 |
| ykarel | thx frickler tkajinam mnasiadka | 09:49 |
| mnasiadka | Thanks tkajinam, was planning to do that now :) | 09:50 |
| tkajinam | mnasiadka, thank you for fixing it :-) | 09:52 |
| *** haleyb|out is now known as haleyb | 13:02 | |
| JayF | docs.openstack is hurting again :( sometimes I wonder if I'm getting caught in glue traps being setup for AI bots | 20:34 |
| fungi | JayF: i think the server is caught in a trap set by ai bots | 20:36 |
| fungi | we're continuing to try to tune apache and waf rules to block the worst offenders | 20:36 |
| mnaser | this is probably not going to be the hottest take, but what about setting up cloudflare for these things, takes the headache out of handrolling all of this stuff | 20:37 |
| fungi | i'm open to the possibility, but we will be accepting that it's currently impossible to run a website on the internet today without handing over all our user data to a compamny who wants to resell it to the highest bidder | 20:39 |
| fungi | also admitting that you can no longer operate a website with open source software | 20:39 |
| mnaser | i think the space is moving so fast and things are hurting a bit, it could be something to give us some breathing space as the oss world figures things out | 20:40 |
| mnaser | i think there's alot of things we can explore such as nginx for example which historically has been far more powerful and resource efficent than apache to serve static content, etc | 20:40 |
| JayF | mnaser++++ took the words outta my mouth | 20:40 |
| JayF | I'd rather have things working and you all free to do things other than tune apache | 20:41 |
| JayF | than stick to an ideal that clearly we're not able to fund | 20:41 |
| fungi | at the moment, it appears that some organized crime outfit in control of hundreds of thousands of backdoored mobile devices has decided to flood every website with random url guesses to see if they can find anything that's not already been indexed normally and fed into existing llm models | 20:41 |
| mnaser | afaik there are things like https://github.com/TecharoHQ/anubis for example, but again deploying this takes time | 20:42 |
| mnaser | heck, they even say you should get cloudflare and this is only if you really dont want to use it =) | 20:42 |
| JayF | fungi: How trust-based is the infra team? We have infra resources at GR-OSS I could potentially ask if they could help e.g. deploy anubis, but I think it'd be a ... project-based interaction, not a long-term-ongoing devops commitment | 20:43 |
| fungi | our systems administration is done entirely in the open through code review, so there's no trust concerns, they wouldn't need the ability to ssh into servers | 20:45 |
| fungi | just pushing (mostly ansible) changes into gerrit | 20:45 |
| JayF | ack | 20:45 |
| JayF | no promises but I can ask | 20:45 |
| fungi | and we're not exactly rolling our own solutions to this stuff, it's just that e.g. traditional waf approaches like apache mod_security aren't entirely suited to the latest wave we've seen over the past week-ish | 20:46 |
| fungi | previously it was crawlers following links from pages to other pages, but now most of the requests the server is handling are for randomly-guessed nonsexitent urls, and not even repeated ones, so things like 404 caching don't help because the server has never seen the request and has to check anyway | 20:48 |
| fungi | urls like | 20:49 |
| fungi | https:///developer/ironic/webapi/contributor/contributor/contributor/contributor/support/admin/drivers/install/admin/cli/contributor/support/contributor/admin/user/contributor/cli/contributor/admin/admin/install/refarch/contributor/api/contributor/admin/contributor/contributor/contributor/contributor/admin/drivers/contributor/admin/admin/install/configuration/policy.html | 20:49 |
| fungi | er, | 20:49 |
| fungi | https://docs.openstack.org/developer/ironic/webapi/contributor/contributor/contributor/contributor/support/admin/drivers/install/admin/cli/contributor/support/contributor/admin/user/contributor/cli/contributor/admin/admin/install/refarch/contributor/api/contributor/admin/contributor/contributor/contributor/contributor/admin/drivers/contributor/admin/admin/install/configuratio | 20:49 |
| fungi | n/policy.html | 20:49 |
| fungi | whatever bot this is seems to take the path components of existing nearby urls and recompose/repeat them in every permutation | 20:50 |
| fungi | i can only guess it's in hopes of finding pages that are otherwise unknown | 20:50 |
| clarkb | I think it is important to call out that all of this is publicly managed and tested pre merge. If people feel strongly that nginx would help (I'm not convinced since it is a ddos effectively) then you can write that change and push it up and start the conversation | 20:52 |
| clarkb | we have also been fairly candid about the mitigation steps we have already taken and those that we think we could use to further alleviate the issue in our matrixroom | 20:52 |
| fungi | "our" being opendev's not the tact sig's | 20:52 |
| clarkb | if people are interested in this stuff then please help otu and get involved. But fungi and I are basically already pretty focused on this and have been since last week and even prior to that (the waf stuff we're doing is preexisting work that started over the holidays) | 20:52 |
| clarkb | its really easy to show up and say You should just use cloudflare or that we're using the wrong webserver | 20:53 |
| clarkb | its actually really not easy to keep all of these services up and running under an onslaught of poorly behavior bots (that know they are poorly behaved and spoof their details) | 20:53 |
| clarkb | all while trying to avoid making the problem worse by over correcting and blocking legit traffic (my phone currently can't talk to github for a concrete exampel of this problem. I think because I have js disabled?) | 20:54 |
| clarkb | this is one of my major concerns with anubis | 20:54 |
| clarkb | particularly for static content that runs zero to minimal js | 20:55 |
| clarkb | (which is the bulk of the content on this particular server) | 20:55 |
| stmcginnis | Long time now see! .o/ | 20:57 |
| clarkb | ftr I don't want to use cloudflare. I think that their business practices around this particular issue are bit scammy (they just added "crawl this site" endpoint). So now rather than being crawled by random people on the internet you're getting crawled by the people you are paying to protect you. | 20:57 |
| stmcginnis | Wanted to make sure someone here saw that http redirects to https are no longer working: https://www.reddit.com/r/openstack/comments/1ruupac/opesntack_docs_down/ | 20:58 |
| fungi | it's not the redirects, it's just that apache is taking a while to process the request regardless of whether it's going to redirect or not | 20:59 |
| fungi | redirects will take longer though, because the client then has to wait for the redirect before requesting the actual page it's being redirected to | 21:00 |
| clarkb | right the problem is a ddos. It isn't specific to things like redirects | 21:01 |
| fungi | clarkb: yes, and probably the biggest upshot of all of this is that even if i was mildly curious about trying out some "ai" based tools before, i have absolutely zero interest in supporting anything to do with the obnoxious cesspool that the ai craze has turned into, burning the internet to the ground in order to try to scam a few more bucks out of people | 21:02 |
| fungi | llms are a menace, plain and simple. their very existence has created perverse incentives to tragedy-of-the-commons everything in sight with absolutely no concern for what collateral damage the do along the way | 21:03 |
| fungi | it's like we gave an nuclear arsenal to a bunch of children and then asked them to play nicely | 21:05 |
| clarkb | it is worth noting that many of the big players seem to identify themselves correctly and don't create these situations. Unfortunately, other crawlers spoof user agents, come from botnet looking diverse ip ranges, request content that doesn't exist, and so on. Its these that are particularly problematic. Blocking google, openai, anthropic, meta, et al is unlikely to help much | 21:05 |
| clarkb | this is where tools like anubis may help by forcing the bots to do proof of work before they can fetch the data. Unfortunately, they require your client to do the same | 21:06 |
| clarkb | if someone wants to deploy anubis and we see how bad it is for users and whether or not it helps against these particular botnets I am not opposed | 21:06 |
| fungi | sadly, the "legitimate" llm operators have also been caught buying training data from dubious sources, perpetuating this stripmine-the-planet goldrush | 21:06 |
| clarkb | it is sad that non js content would require js just to filter out the bots though | 21:06 |
| fungi | also tomorrow's llm training crawlers will just throw in a client-side js processor to work around anubis, but hey maybe it buys us a few days before we have to replace it with yet-something-else | 21:08 |
| clarkb | yes, I'm honestly surprised they haven't done this yet. I Think tehy can share the cookie even so you don't have to calculate it over and over again | 21:08 |
| fungi | my guess is that the only reason they haven't so far is that so little of value to them on the internet puts a solution like that in place, so it's cheaper to just skip/ignore whatever's there | 21:09 |
| fungi | but as it becomes more popular, there will be an increasing incentive to work around | 21:09 |
| fungi | and by putting our eggs in that basket, we're increasing that incentive | 21:10 |
| clarkb | thinking out loud about the patterns we've seen for docs particularly I wonder if some sort of rate limiting mehcanism would be a good idea | 21:17 |
| clarkb | though possibly easily defeated by the many many ip addresses | 21:17 |
| fungi | yes, i'll run some numbers, but probably better to discuss in #opendev:opendev.org matrix | 21:19 |
| clarkb | ++ moving there | 21:19 |
| JayF | My big concern is just we have hundreds of people who are slowed down by these issues; they aren't the fault of OpenDev but we need to shine more light on them so we can resolve them. It's going to drive away users and contributors if it hasn't already. | 21:27 |
| JayF | I am not trying to proscribe any solution, just indicating that no solution is existentially threatening to openstack | 21:27 |
| fungi | hopefully we can get the word out that trying to operate services supporting multi-million dollar companies on a semi-volunteer skeleton crew is a risk, and that some additional investment in the way of more actual sysadmins would go a long way to mitigating it. not when things are broken and falling down around us and we have to time to train anyone up, but during the calm | 21:29 |
| fungi | times when it's running smoothly | 21:29 |
| clarkb | and it goes beyond just opendev. If you're talking existential threats to openstack I could probably write up a list a mile long. But starting where there is acute pain is probably a good starting point | 21:31 |
| fungi | we have a couple of close-to-full-time people, a couple of part-time people, and another part-time person being trained up to help | 21:31 |
| clarkb | re nginx it was pointed out that openstac documentation currently relies heavily on htaccess files for redirects | 22:00 |
| clarkb | so that won't work wtihout some conversion to nginx redirects or similar | 22:00 |
| fungi | yes, more to the point it's not just that it relies on .htaccess, but has a bunch of bespoke tooling and associated test framework for generating templated .htaccess rules and making sure they're not broken | 22:02 |
| mnaser | clarkb: You can just opt out of it. At some point we need to stabilize the infra here.. | 22:08 |
| clarkb | sure that is an option too | 22:09 |
| clarkb | I just want to call it out as "this will berak and opesntadck will be broken in a different way unless this is handled too" for the nginx idea | 22:09 |
| clarkb | I don't thinka nyone is arguing about that. But what is missed is that fungi and I have been actively working on this specific problem since last week. And actively dealing with the dramatic change in internet traffic for months | 22:10 |
| clarkb | today everyone notices because the mitigations are not sufficient. But no one is aware of the other ddos storms that we've managed to deal with | 22:10 |
| clarkb | this isn't new or foregin to us and it has been something that has eaten signficiant amount of time. Taking time away from other important tasks. This is why more help would be great | 22:11 |
| fungi | yes, the pattern of abuse has changed significantly this time, and the various solutions we've had in place weren't designed for this particular case so it's taking some new engineering | 22:11 |
| clarkb | it doesn't help that the openstack docs team which would be responsible for helping us disolved years ago and the TC which took on those duties has basically ignored them | 22:12 |
| fungi | the docs team used to develop their own site management tooling, even | 22:12 |
| clarkb | we're what is left and we're doing our best. If that isn't good enough we'd really appreciate some help | 22:13 |
| fungi | but as people leave, responsibility gets increasingly concentrated on the ever fewer who stick around, until it burns them out too | 22:13 |
| fungi | these tasks have not traditionally been well-staffed because this is a volunteer project and services are maintained on a best-effort basis, but sometimes our best effort isn't enough to avoid a bit of pain | 22:14 |
| fungi | we make up for and paper over a lot of it through automation, so it often goes unnoticed how small of a group of people we really have keeping it all running | 22:15 |
| JayF | clarkb: I told fungi this in DM, I'll say it here so you can hear: part of why I'm raising the alarm is it's *obvious* you all have been doing heroic things to keep it online. That is extremely appreciated ... and not sustainable. There need to be solutions that don't involve the infra team burning themselves out :( (I hope they exist) | 22:42 |
Generated by irclog2html.py 4.1.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!