| JayF | Has something happened recently which would've caused security.openstack.org links to get massively deindexed by google? | 17:30 |
|---|---|---|
| JayF | specifically, e.g. https://security.openstack.org/vmt-process.html isn't showing up in google anywhere even when searching specifically for its keywords | 17:30 |
| clarkb | it is also hosted on the same system as docs.openstack.org. Is it possible that when things were falling over for docs.openstack.org (which also affected the other names hosted there) that created problems? We aren't actively preventing Google from indexing them (and no anubis either) we went with make bigger server as the solution which is what we're currently doing | 17:32 |
| clarkb | if you do site:security.openstack.org does google still return results for it? | 17:33 |
| clarkb | (that would imply it knows about it but isn't prioritizing it for some reason) | 17:33 |
| JayF | it returns a subset of pages | 17:35 |
| JayF | and is missing most of the primary nav pages; the OSSA-xxxx-xxxx are still there | 17:35 |
| clarkb | fwiw (and I have no idea if these things are related) I've found taht google search has gotten really bad about pushing official documentation for a lot of things. I commonly hit this for python | 17:36 |
| JayF | Yeah, I think there's a good chance it's just Google getting bad, but I wanted to mention it in case something had been changed recently in the AI mitigation efforts. | 17:40 |
| clarkb | I don't think so unless it was the actual overloaded system itself that side effected this by not being available for the google indexers when we had issues | 17:41 |
| JayF | that'd makes sense, too | 17:42 |
| clarkb | the only site we're running anubis on right now is lists and we believe our anubis configuration should allow google though. We also have some user agent fingerprinting rules but those don't block google either | 17:42 |
| clarkb | I suppose it is possible that google hit our honeypot and got blocked. But google didn't show up in the honeypot hits when I checked it last (it was a really small number of things) | 17:43 |
| clarkb | and I don't think the honeypot is running anymore | 17:43 |
Generated by irclog2html.py 4.1.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!