| @clarkb:matrix.org | I have discovered that thunderbird doesn't allow you to turn off notifications on a per account basis without an extension. Thats ok I don't need any notifications for now | 15:20 |
|---|---|---|
| @clarkb:matrix.org | infra-root https://review.opendev.org/c/opendev/system-config/+/979874 and child are updates to gerrit testing and image building that shouldn't affect production (because the 3.11 image and tag aren't modified). Both are helpful for preparing eventual Gerrit upgrades. | 15:27 |
| @clarkb:matrix.org | the giteas look like they are getting crawled again | 17:04 |
| @tafkamax:matrix.org | yeah its slow to browse | 17:05 |
| @clarkb:matrix.org | yes, I'm trying to characterize it now | 17:07 |
| @clarkb:matrix.org | then hopefully be able to push back | 17:07 |
| @tafkamax:matrix.org | Is there any info on devstack recommended specs for cpu, ram, storage. I tried a simple search but did not find that info. | 17:11 |
| @fungicide:matrix.org | Taavi Ansper: upstream openstack test jobs run devstack on our standard 8gb ram vm flavors: https://docs.opendev.org/opendev/infra-manual/latest/testing.html | 17:12 |
| @fungicide:matrix.org | 4-8vcpu depending on how fast the hosts are, 80gb rootfs (or 40gb rootfs with 80gb in /opt) | 17:13 |
| @tafkamax:matrix.org | ok, that is another place to search indeed. I was lookin in devstack docs here: https://docs.openstack.org/devstack/latest/index.html | 17:14 |
| @fungicide:matrix.org | it's hard to give an exact answer to the question because it depends a lot on which services you enable/disable in devstack and what operations you perform | 17:15 |
| @fungicide:matrix.org | if it's not in a testing context you may want a lot more resources too, for the upstream testing example jobs mostly just boot nested cirros images for guests and don't do much more than ssh into them and run simple commands to verify they're reachable/functional | 17:16 |
| @tafkamax:matrix.org | I am trying to debug new version horizon and heat-plugin. https://review.opendev.org/c/openstack/kolla-ansible/+/979962?tab=change-view-tab-header-zuul-results-summary | 17:16 |
| @tafkamax:matrix.org | heat-dashboard plugin* | 17:16 |
| @fungicide:matrix.org | Taavi Ansper: do any of those jobs use devstack? i think they're deploying with kolla instead | 17:17 |
| @fungicide:matrix.org | kolla-ansible rather | 17:18 |
| @tafkamax:matrix.org | I was just thinking I could get an UI up wth the dashboard enabled using devstack for the latest version. | 17:18 |
| @tafkamax:matrix.org | And if that doesn't give me anything interesting I will do a kolla install probably. | 17:19 |
| @fungicide:matrix.org | yeah, if you just want the webui up with heat and other basic apis, an 8gb vm with a handful of vcpus and 80gb rootfs should be plenty, but you could probably fgo even smaller | 17:22 |
| @fungicide:matrix.org | also be aware that we (the opendev community) don't maintain devstack, so you might have more luck talking to the devstack maintainers in the #openstack-qa channel on oftc irc | 17:23 |
| @tafkamax:matrix.org | ok thx for info, will join there | 17:29 |
| @jim:acmegating.com | I think this will make a nice friendly clickable matrix link: #_oftc_#openstack-qa:matrix.org | 18:13 |
| @jim:acmegating.com | * I think this will make a nice friendly clickable matrix link to that channel via the oftc-matrix bridge: #_oftc_#openstack-qa:matrix.org | 18:14 |
| -@gerrit:opendev.org- Clark Boylan proposed: [opendev/system-config] 980144: Add large blocks of UA filters https://review.opendev.org/c/opendev/system-config/+/980144 | 18:19 | |
| @clarkb:matrix.org | I hate that ^ is a thing but I think that is the next step. I am slightly worried that my regexes may over match. I'm trying to do things like match singel and double digit versions allowing triple digit through so reviews may want to look at that part of the regexes more closely | 18:20 |
| @clarkb:matrix.org | fungi: then fungi I was thinking there are a number of UAs that look more modern and thus may be valid. So a good followup would be adding rules like this to the mod security for gitea. The problem is we're terminating where we don't have great source ip address info on the backend so may need to figure that out first | 18:21 |
| @fungicide:matrix.org | yeah, i wonder if apache sticks the forwarded-for protocol field that haproxy adds into a var we can reference with mod_security? | 18:23 |
| @clarkb:matrix.org | that is a good question. I was worried that wouldn't work because gitea doesn't handle x forward for properly. But we'd be doing this a level before gitea so gitea's brokenness doesn't matter. | 18:23 |
| @clarkb:matrix.org | so ya as long as haproxy is supplying the info (I think it is) we can theoretically start comparing those x forwarded for values and use that in the block db and the db lookups | 18:24 |
| @clarkb:matrix.org | nhicher: https://review.opendev.org/c/opendev/system-config/+/980144 is the change I mentioned for pushing back against the web crawl making opendev.org slow | 18:40 |
| @nhicher:matrix.org | ok, thanks | 18:41 |
| @fungicide:matrix.org | which is now approved, but will take some time to deploy still | 18:41 |
| @clarkb:matrix.org | its worth noting that update is also probably incomplete. But I'm hopeful it will have a positive impact and then we can continue to add large blocks of problematic users to block lists one way or another | 18:42 |
| @fungicide:matrix.org | is this randomized url paths again like we saw hitting docs.openstack.org last week, or more typical link following just very rapidly/in parallel? | 18:44 |
| @clarkb:matrix.org | fungi: I didn't see obvious randomized url paths. Instead its the classic crawl every file for every commit in every git repo problem | 18:45 |
| @clarkb:matrix.org | but if you want to take a look at gitea access logs and double check I didn't miss something that would be great too | 18:45 |
| @clarkb:matrix.org | Another thing we could look at is whether or not it makes sense to increase the memcached cache size, but usually this every file for every commit in every repo behavior is a cache destroyer since it isn't touching hot data over and over and instead just thrashing the cache with info that will not be rerequested ever again | 18:47 |
| @fungicide:matrix.org | yep, jumping on 09 in a sec | 18:47 |
| @fungicide:matrix.org | yeah, the only things increasing memcached space will likely do are make pulling from the cache slower and give it a little more time before the cache fills up with randomness causing most everything to be a cold hit | 18:48 |
| -@gerrit:opendev.org- Clark Boylan proposed: [opendev/system-config] 980144: Add large blocks of UA filters https://review.opendev.org/c/opendev/system-config/+/980144 | 18:58 | |
| @clarkb:matrix.org | the lists job failed and complained about a bad regex. I think ^ fixes it | 18:58 |
| @fungicide:matrix.org | ah, yep, a couple of missing backslash escapes | 19:14 |
| @clarkb:matrix.org | I've managed to catch the check node for gitea testing 980144 and can browse a few pages and I don't get blocked so I probalby haven't broken everyone with this ruleset update | 19:40 |
| @clarkb:matrix.org | as a heads up I've purchased one last annual cert for wiki.openstack.org. This ensures we get ahead of the march 15 change to 200 day max validity increasing the time period between when we would have to do renewals next. The details are in the usual location on bridge and it sounds like fungi will get it applied to the server | 21:08 |
| @fungicide:matrix.org | yep, probably in an hour-ish | 21:12 |
| -@gerrit:opendev.org- Zuul merged on behalf of Clark Boylan: [opendev/system-config] 980144: Add large blocks of UA filters https://review.opendev.org/c/opendev/system-config/+/980144 | 21:18 | |
| @clarkb:matrix.org | gitea, lists, static, and zuul will be affected by ^ as mentioned I did some sanity checking against the running test instance while it was up and it looked ok. But be on the lookout for unexpected problems | 21:19 |
| @clarkb:matrix.org | I'll check things once deployments are done | 21:19 |
| @clarkb:matrix.org | all three of my browsers can still browse docs.opendev.org | 21:23 |
| @clarkb:matrix.org | and system load is falling a bit on gitea09 | 21:23 |
| @fungicide:matrix.org | yeah, still working for me | 21:24 |
| @clarkb:matrix.org | nhicher: hopefully things are better (for) now. The mitigation seems to be working based on system load metrics | 21:32 |
| @nhicher:matrix.org | Clark: thanks, I will check on our side | 21:33 |
| @nhicher:matrix.org | I still get: 'Job tox not defined' for zuul can't load zuul-jobs | 21:39 |
| @nhicher:matrix.org | I will try to reconfigure the tenant | 21:39 |
| @fungicide:matrix.org | if zuul timed out during configuration, then yes it may not know it can check again unless you tell it | 21:40 |
| @clarkb:matrix.org | looks like Zuul doesn't consistently override its user agent in the various connection drivers. Otherwise I would be able to grep off of that easily | 21:42 |
| @clarkb:matrix.org | the gerrit connection does but doesn't look like any other do | 21:43 |
| @clarkb:matrix.org | anyway I shouldn't have created any rules that affect requests or git either so I doubt this is the issue | 21:43 |
| -@gerrit:opendev.org- Ivan Anfimov proposed wip: [openstack/project-config] 980169: Update description for retired project (ironic-inspector) https://review.opendev.org/c/openstack/project-config/+/980169 | 21:47 | |
| @nhicher:matrix.org | fungi, Clark it's good for us now -> Starting check jobs | 21:47 |
| @nhicher:matrix.org | thanks =) | 21:47 |
| @clarkb:matrix.org | great | 21:47 |
| -@gerrit:opendev.org- Ivan Anfimov proposed wip: [openstack/project-config] 980169: Update description for retired project (ironic-inspector) https://review.opendev.org/c/openstack/project-config/+/980169 | 21:48 | |
| @nhicher:matrix.org | for our cgit, we use anubis, it works really well, I don't know if it can work for you, our configuration is really simple https://softwarefactory-project.io/cgit/software-factory/sf-infra/tree/roles/service/anubis/templates/botPolicies.yaml.j2 | 21:49 |
| @clarkb:matrix.org | I have been hesitant to use anubis because I personally find the user experience to be well terrible | 21:49 |
| -@gerrit:opendev.org- Ivan Anfimov proposed wip: [openstack/project-config] 980169: Update description for retired project (ironic-inspector) https://review.opendev.org/c/openstack/project-config/+/980169 | 21:50 | |
| @clarkb:matrix.org | and in theory it is easily defeatable by the bots by simply calculating the hash then reusing the cookie | 21:50 |
| -@gerrit:opendev.org- Ivan Anfimov proposed wip: [openstack/project-config] 980169: Update description for retired projects https://review.opendev.org/c/openstack/project-config/+/980169 | 21:50 | |
| -@gerrit:opendev.org- Ivan Anfimov marked as active: [openstack/project-config] 980169: Update description for retired projects https://review.opendev.org/c/openstack/project-config/+/980169 | 21:50 | |
| @clarkb:matrix.org | I'm not opposed to anubis if we end up in a situation where nothing else is working. But for now we've been able to keep things going with some simple web server rules | 21:51 |
| @nhicher:matrix.org | sure, for our usage it works really well, we reduced the load from 200 to 0.7 =) | 21:52 |
| @clarkb:matrix.org | nhicher: one thing to note if you look at my mitigation today is that the UAs are no longer limited to Mozilla. Though the ratio is still largely Mozilla UAs. | 21:54 |
| @fungicide:matrix.org | a lot do pretend to be chrome, or safari | 21:54 |
| @clarkb:matrix.org | fungi: well those still report Mozilla. Its opera that doesn't | 21:55 |
| @fungicide:matrix.org | ah yeah | 21:55 |
| @fungicide:matrix.org | i thought you meant actual mozilla-based not merely mozilla-compliant | 21:55 |
| @fungicide:matrix.org | (i read it as s/mozilla/firefox/ really) | 21:56 |
| @clarkb:matrix.org | fungi: anubis keys off the first bit of your User agent to decide if it shoudl challenge the client or not. You can see that here: https://softwarefactory-project.io/cgit/software-factory/sf-infra/tree/roles/service/anubis/templates/botPolicies.yaml.j2#n16 This means that you both catch well behaved bots in the challenges (like googlebot etc) and don't challenge anyone using a different agent. Today I noticed that opera has picked up stream | 21:56 |
| @clarkb:matrix.org | * fungi: anubis keys off the first bit of your User agent to decide if it shoudl challenge the client or not. You can see that here: https://softwarefactory-project.io/cgit/software-factory/sf-infra/tree/roles/service/anubis/templates/botPolicies.yaml.j2#n16 This means that you both catch well behaved bots in the challenges (like googlebot etc) and don't challenge anyone using a different agent. Today I noticed that opera has picked up steam | 21:56 |
| @clarkb:matrix.org | but its still a relatively low proportion of the problematic traffic so probably not large enough to break things | 21:57 |
| @fungicide:matrix.org | debian ended up adding something anubis-like to all their community web properties across the board. i haven't looked into it but basically any time i pull up a something.debian.org page i get a brief "calculating" javascript widget before it loads | 21:57 |
| @nhicher:matrix.org | Clark: yes, it's a never ending story with bots | 21:58 |
| @clarkb:matrix.org | fungi: I'm surprised debian would do that given it requires you to run js to make things work. Seems like they would be at the forefront of not requiring js at all. But amybe the current state of the Internet has forced their hand | 21:58 |
| @fungicide:matrix.org | i suspect they only do it for user agents that look like javascript-capable browsers | 22:02 |
| @fungicide:matrix.org | #status log Manually rotated the wiki.openstack.org SSL certificate updating from 2026-04-29 to 2027-03-11 expiration | 22:14 |
| @status:opendev.org | @fungicide:matrix.org: finished logging | 22:15 |
| @fungicide:matrix.org | statusbot also served as a functional test that the cert rotation didn't break its mediawiki api integration | 22:15 |
| @clarkb:matrix.org | the new cert looks good in firefox as well | 22:43 |
| @fungicide:matrix.org | thanks for testing! | 22:43 |
| @fungicide:matrix.org | and for buying/expensing the cert | 22:43 |
| @clarkb:matrix.org | yes I need to go figure out what the expensing process is for this now. | 22:45 |
| @fungicide:matrix.org | okay, thanks *in advance* then ;) | 22:45 |
| @clarkb:matrix.org | mnasiadka: I didn't get around to approving your user account and ssh key change today. At this point its late enough that I probably won't do it and instead will aim for tomorrow morning. The crawlers hitting gitea really grabbed my attention | 22:46 |
| @fungicide:matrix.org | the crawlers grab a lot of things, that's the problem | 22:46 |
Generated by irclog2html.py 4.1.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!