| mnasiadka | I think somebody took the approach of small patches easy to review too much into his heart :) | 05:21 |
|---|---|---|
| *** ykarel__ is now known as ykarel | 08:01 | |
| frickler | I really like this stack, even if it puts zuul to a test. also the good news is that the kolla CI seems to be very stable, seems there was only a single unrelated failure | 08:32 |
| clarkb | ya I think its fine to exercise zuul. That said I think linter updates like that are often better done in one go that way you don't have half the codebase in one state and another half in another while slowly backsliding against the rule you're trying to fix | 15:48 |
| clarkb | it did seem to confuse my firefox browser and now it keeps thinking zuul wants to open pop ups | 15:50 |
| clarkb | looks like it is trying to open change links. I wonder if I managed to accidentally click on a bunch when the browser was chugging away at processing hte page or something | 15:51 |
| clarkb | corvus: after some breakfast I'm going to create teh statusbot account finally. I notice in the statusbot code update for matrix support your example has password=PASSWORD. I think matrix will give us both an account password and a token? Are they interchangeable here or is the password specifically what we need? | 15:54 |
| corvus | clarkb: i think it wants the password (which it uses to get tokens)... possible we could add a token support at some point? only 80% confidence in this answer. | 15:58 |
| clarkb | got it thanks | 15:58 |
| clarkb | `Failed to add user - MAS GraphQL operation AddEmail failed. Expected status ADDED, got DENIED` | 16:19 |
| clarkb | however, the user account shows up in the user list | 16:19 |
| clarkb | if I pull it up the email address is blank like the other accounts. Weird | 16:19 |
| clarkb | I was then able to reset the password to a value I know. So I think I'm just going to roll with it | 16:21 |
| clarkb | I stored the account info in the typical location and then updated private ansible vars so https://review.opendev.org/c/opendev/system-config/+/969328 and parent should also be good to go now | 16:33 |
| clarkb | as for the email address add operation failure my best guess is that they are mistaken that plus addressing is supported or that using a plus address shared with the top level management account's email address doesn't count as unique enough | 16:34 |
| clarkb | fungi: are there rename prep changes yet? I'm trying to remember the process. Its basically have change with the data structure to input to the rename playbook and also general records keeping info. Then we put services in the emergency file, run the playbook with that input, then we land the change to reflect that in project-config gerrit/projcets.yaml, then we clean up emergency | 16:44 |
| clarkb | file (possibly before landing the prior change to check it noops) and we land our records keeping change? | 16:44 |
| fungi | https://review.opendev.org/970307 | 16:45 |
| clarkb | thanks that change lgtm when compared against its depends on | 16:48 |
| clarkb | it is probably worth a quick read through of the rename playbook just to ensure none of the tasks have bitrotted away (our testing of it only covers its execution against gerrit) | 16:50 |
| clarkb | for example I think we rename the zuul secrets. I assume that command still works | 16:50 |
| fungi | https://etherpad.opendev.org/p/opendev-project-renames-20240422 looks like the last time we did it? | 16:52 |
| clarkb | Pretty sure we didn't do any in 2025 so that seems likely | 16:53 |
| clarkb | fyi more docs.openstack.org woes though I've asked for florian to confirm it is since we bumped up server limits https://lists.openstack.org/archives/list/openstack-discuss@lists.openstack.org/thread/SRGBA7QFQEEAKQMSOTP35AYJDXZ2GZA4/ | 17:20 |
| clarkb | we're currently at about 70% of capacity | 17:20 |
| fungi | i still like the honeypot idea. create a tripwire url that we don't link from other documents, add a disallow rule for it in our robots.txt, then inject short-term block rules for any ip address that requests it | 17:33 |
| fungi | curious if anyone's seen a solution like that employed in the wild yet | 17:34 |
| clarkb | fungi: yes there is a tool to do it too but the author indicates they think it may haev legal implications for some reason | 17:35 |
| clarkb | I do think that is likely to be more effective than anubis though | 17:35 |
| clarkb | but you may need to sprinkle honeypot changes over time | 17:35 |
| fungi | if you're talking about the one i'm thinking of, that directs the abuser to a malicious stream of tarpitted nonsense output rather than just blocking their ip address | 17:35 |
| fungi | so i can understand why it may be of dubious legality | 17:36 |
| clarkb | oh ya that is the one and ya maybe that is the difference that is problematic | 17:36 |
| clarkb | fungi: what would that look like with our current apache2 based system on static.o.o with afs backed content. Put a specific file rule in each vhost for something like info-uuid4.html and then how do you track who requests that to block them? | 17:42 |
| clarkb | would we need to write a small wsgi server or something like that to track the requests and filter them? I assume apache has some sort of mechanism for doing this | 17:42 |
| clarkb | essentially we need some way of maintaining a database of requests to the honeypot and a way for apache2 to check that database before deciding if requests can proceed to other data | 17:43 |
| fungi | yeah, i was hoping there might already be some sort of existing application that does it, otherwise would require some development | 17:43 |
| fungi | though fail2ban could probably be tuned to do it | 17:44 |
| fungi | just watch the access log, then insert nft rules | 17:44 |
| fungi | it wouldn't work as well behind haproxy though, so e.g. gitea couldn't take advantage of it | 17:45 |
| clarkb | we could run it on the haproxy frontend though | 17:45 |
| clarkb | the other issue with that approach is all the nat users we have | 17:45 |
| clarkb | ideally we'd be filtering by more than just ip address | 17:45 |
| clarkb | but maybe we're to the point where that is what we can do so we do it | 17:45 |
| fungi | well, problem with running on gitea-lb is that we don't terminate ssl/tls there | 17:46 |
| fungi | so that machine doesn't know what urls clients are hitting | 17:46 |
| fungi | and yes, an abuser running some bot behind a nat that also has legitimate users can lead to blocking them as collateral damage, but most other services out there on the internet seem to accept that risk already | 17:47 |
| fungi | if some shady bot running behind red hat's corporate nat crawls content we've explicitly disallowed in robots.txt, then that'll provide some pretty quick feedback to folks there, for example | 17:49 |
| clarkb | ya I'm more worried about NAT happenign at the isp level | 17:50 |
| clarkb | https://www.haproxy.com/blog/use-haproxy-response-policies-to-stop-threats haproxy could do it for us | 17:52 |
| clarkb | I think you'd use stick table entries to record who is requesting the honeypot then deny all requests for entries in that table | 17:52 |
| clarkb | (but I'm not sure if you can add entries there based on request path) | 17:52 |
| fungi | but yeah, we'd have to switch to terminating https in haproxy | 17:54 |
| fungi | which might be a little more reasonable these days, since we started blocking direct requests to the backend servers anyway | 17:55 |
| clarkb | mod_security is another option looks like but I can't find configuration docs on what it supports doing | 17:57 |
| clarkb | yes looks like mod security supports tables that we can then modify based on conditions | 17:59 |
| clarkb | so that is probably the easiest solution with our existing apache vhost setup if someone is able to figure out the syntax to do so | 18:00 |
| clarkb | https://github.com/owasp-modsecurity/ModSecurity/wiki/Reference-Manual-%28v2.x%29-Configuration-Directives#secaction initcol is the persistent collection/table action | 18:03 |
| clarkb | then you have a SecRule that adds entries to the collection/table if the request url is the honeypot path. Then you Have another SecRule that blocks requests if the ip/requestor is in that collection/table | 18:04 |
| clarkb | and then you can set things up to expire out of the table after some time period | 18:05 |
| fungi | yeah, i've used it before, but it's been well over a decade | 18:11 |
| fungi | gonna go grab some lunch, bbiab | 18:11 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Add modsecurity waf rules to docs.openstack.org https://review.opendev.org/c/opendev/system-config/+/970674 | 18:44 |
| clarkb | I don't expect that to work on a first pass | 18:44 |
| clarkb | but I think the general shape is there if others want to weigh in on the approach | 18:44 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Add modsecurity waf rules to docs.openstack.org https://review.opendev.org/c/opendev/system-config/+/970674 | 19:14 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Add modsecurity waf rules to docs.openstack.org https://review.opendev.org/c/opendev/system-config/+/970674 | 20:01 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Add modsecurity waf rules to docs.openstack.org https://review.opendev.org/c/opendev/system-config/+/970674 | 20:50 |
| clarkb | the config loads now but testing shows it doesn't work as expected. That last patchset is more ah unch than anything else after noticing the reference manual using lower case references for the collection when initing it and modifying its attributes | 20:51 |
| clarkb | if that doesn't fix it the next step is to update the rules with log entries so that the apache error log contains info on what is happening | 20:51 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Add modsecurity waf rules to docs.openstack.org https://review.opendev.org/c/opendev/system-config/+/970674 | 21:26 |
| clarkb | apparently the rule needs to be =+1 not +=1 | 21:26 |
| clarkb | I'm somewhat surprised it parsed ok prevuously though so maybe this isn't it | 21:26 |
| clarkb | I'm going to pop out for a bike ride now. Feel free to update ^ if anyone else is interested in making that work | 21:26 |
| mnaser | lol, looks like opendev isn't the only one struggling with scrapers.. evn at the bigger scales - https://www.githubstatus.com/incidents/40730vhmg6y8 | 23:34 |
| fungi | lovely | 23:42 |
Generated by irclog2html.py 4.0.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!