| @mnaser:matrix.org | when i curl i get anubis'd .. | 00:00 |
|---|---|---|
| @mnaser:matrix.org | but when i look at the source though | 00:00 |
| @mnaser:matrix.org | ``` | 00:00 |
| <script id="anubis_challenge" type="application/json">{"rules":{"algorithm":"fast","difficulty":4},"challenge":{"issuedAt":"2026-08-05T23:59:41.969945208Z","metadata":{"User-Agent":"curl/8.5.0","X-Real-Ip":"38.108.68.97"},"id":"019fd45e-5d91-7e4b-b3a0-6c361d0d5a57","method":"fast","randomData":"971f1e363fb16533d4f514f57763ac810552f4e484bb3fc9a46aa4aa9e7e986b1f0e1468f7efab761111d63dfe4efd1576aafc7f0cf60186e5c4ea3622fba07a","policyRuleHash":"ac980f49c4d35fab","difficulty":4,"spent":false}} | ||
| </script> | ||
| ``` | ||
| @mnaser:matrix.org | X-Real-Ip .. thats one of our vexxhost ips.. i suspect thats an haproxy ip | 00:00 |
| @clarkb:matrix.org | yes we have haproxy in front of 6 gitea backends | 00:01 |
| @clarkb:matrix.org | you shouldn't be getting anubis with curl or git (I don't) | 00:01 |
| @mnaser:matrix.org | shouldn't x-real-ip be my ip address rather than the anubis one? | 00:01 |
| @mnaser:matrix.org | or something on the remote side so to speak | 00:01 |
| @mnaser:matrix.org | that X-Real-Ip is the IP address of gitea-lb03.opendev.org | 00:02 |
| @clarkb:matrix.org | anubis is behind the haproxy so I think it is seeing the haproxy ip as we don't have PROXY protocol enabled | 00:03 |
| @clarkb:matrix.org | (that is in the works but there is no other l3/l4 method for telling the backend that there is a client further back) | 00:03 |
| @mnaser:matrix.org | im hitting gitea14.opendev.org from the env that is failing, let me try to pin request locally | 00:04 |
| @mnaser:matrix.org | unless that's not possible anymore | 00:05 |
| @clarkb:matrix.org | it isn't | 00:05 |
| @clarkb:matrix.org | we had to block direct access due to all of the crawling (it was better to get crawled once per botnet rather than 7 x per botnet) | 00:06 |
| @clarkb:matrix.org | and yes I'm looking at the logs on gitea14 to try and figure it out too | 00:06 |
| @mnaser:matrix.org | failing system => User-Agent: curl/8.5.0 | 00:06 |
| my system => User-Agent: curl/8.21.0 | ||
| @clarkb:matrix.org | I see it in the logs for both your git and curl (I think you must've manually run curl?) | 00:10 |
| @mnaser:matrix.org | yes i've been trying to prod at it from the CI node with git and curl too | 00:10 |
| @clarkb:matrix.org | and it does say issue challenge. So maybe something about the upgrade today got us some new dynamic rules that we weren't expecting | 00:10 |
| @clarkb:matrix.org | our actual config is straightforward and dead simple and basically if it is a browser user agent you get a challenge other wise you're good | 00:10 |
| @mnaser:matrix.org | i wonder if its doing something based on ip range now | 00:11 |
| @clarkb:matrix.org | if it was I would expect all of our requests to fail due to the proxy | 00:11 |
| @clarkb:matrix.org | so I don't think it is that | 00:11 |
| @mnaser:matrix.org | changing my user agent locally is still working | 00:11 |
| @mnaser:matrix.org | ah right cause it doesnt know the right ip | 00:11 |
| @mnaser:matrix.org | Clark: i found this commit -- https://github.com/TecharoHQ/anubis/pull/1521/changes -- diggin deeper since thats doc change only | 00:15 |
| @clarkb:matrix.org | I think that is the documentation for having haproxy as a proper http proxy in our case we have apache in that role | 00:16 |
| @clarkb:matrix.org | I'm thinking it may be something with how they load the ruleset configs? like maybe there is some default handling I see the weight is pretty high on those requests so I'm wondering if there is a default if weight is too high handle it. | 00:17 |
| @mnaser:matrix.org | is there a list of ips.. maybe i have a bad ip | 00:17 |
| @clarkb:matrix.org | no again because there is only one ip | 00:18 |
| @clarkb:matrix.org | we can't make that work currently | 00:18 |
| @mnaser:matrix.org | ah right! yes sorry, keep forgetting that | 00:18 |
| @clarkb:matrix.org | (anubis can do that but we can't) | 00:18 |
| @clarkb:matrix.org | https://opendev.org/opendev/system-config/src/branch/master/playbooks/roles/gitea/files/anubis_botPolicy.yaml this is the policy we think we're using | 00:19 |
| @mnaser:matrix.org | does anubis not mention why something was challenged? | 00:19 |
| @clarkb:matrix.org | no | 00:21 |
| @clarkb:matrix.org | it logs a challenge uid and the weight and the request but not the rule that triggered it | 00:22 |
| @clarkb:matrix.org | I'm beginning to wonder if the udpated container image is simply not loading our config and using some default config | 00:22 |
| @mnaser:matrix.org | Clark: i think it does because | 00:23 |
| ``` | ||
| curl -v -H "User-Agent: curl/8.5.0" -H "CF-Worker: hey" https://opendev.org/openstack/hacking/info/refs | head | ||
| ``` | ||
| now i dont know if thats a general default rule, but it is one we denyt | ||
| @clarkb:matrix.org | oh are you using cloud flare? | 00:25 |
| @clarkb:matrix.org | then yes we will deny you | 00:25 |
| @mnaser:matrix.org | no i am not! but it was to tes the rule | 00:25 |
| @clarkb:matrix.org | oh yes blocking cloudflare is pretty universal aiui | 00:25 |
| @clarkb:matrix.org | they are a huge source of abuse | 00:25 |
| @mnaser:matrix.org | i am getting a challenge otherwise when attempting to curl from my failing node | 00:25 |
| @mnaser:matrix.org | right yeah i was just trying to sus out if the config is applied or not but it seeems that is not a strong enouh signal | 00:25 |
| @clarkb:matrix.org | looks like the healthchecks are failing and running it manually I get `failed to fetch metrics: parse "http://localhost[::1]:15000/healthz": invalid IP-literal` We don't put that url into our config so it really does seem like there is at least some config mangling happening | 00:28 |
| @clarkb:matrix.org | `METRICS_BIND: "[::1]:15000"` is the config value it should be using | 00:28 |
| @clarkb:matrix.org | if POLICY_FNAME is unset then the default policy is used according to the docs | 00:31 |
| @clarkb:matrix.org | now to figure out if that is set on the process | 00:31 |
| @clarkb:matrix.org | /proc/$PID/environ says it is set | 00:32 |
| @clarkb:matrix.org | the .ko.yaml file hasn't changed in a meaningful way. But I'm also not seeing anywhere obvious that handles POLICY_FNAME | 00:35 |
| @mnaser:matrix.org | Clark: my fellow clanker has potentially root caused this | 00:35 |
| @mnaser:matrix.org | https://github.com/TecharoHQ/anubis/commit/1d700a037087a2a8695d9f7916f356be15f0a8e3 | 00:36 |
| @mnaser:matrix.org | it seems they added something relying on ip addresses and i wonder if the instance of 13 got triggered some how | 00:36 |
| @mnaser:matrix.org | "OpenDev’s policy omits a honeypot setting, and 1.26 defaults it to enabled." | 00:37 |
| @clarkb:matrix.org | I think it was enabled in 1.25 too, but that chagne is new in 1.26 | 00:38 |
| @clarkb:matrix.org | we could try disabling it I guess? | 00:38 |
| @mnaser:matrix.org | https://github.com/TecharoHQ/anubis/blob/main/data/clients/git.yaml | 00:38 |
| we can apparently use this to which is a git allowlist | ||
| @clarkb:matrix.org | you think that since we dno't have an explicit allow its filtering through into some default handling path? | 00:39 |
| @clarkb:matrix.org | I'm not sue I understand why that chagne would affect this fwiw | 00:39 |
| @mnaser:matrix.org | well because it went from using user agent to do its honeypot classification to cidr's | 00:40 |
| @mnaser:matrix.org | and well, we all share the same cidr | 00:40 |
| @clarkb:matrix.org | right but that should cause every single requset to hit this problem | 00:40 |
| @clarkb:matrix.org | but git and curl work fine for me from my local machine | 00:40 |
| @clarkb:matrix.org | oh you know what it is | 00:41 |
| @clarkb:matrix.org | its the weight | 00:41 |
| @mnaser:matrix.org | maybe that is what is happening in gitea13 only? | 00:41 |
| @clarkb:matrix.org | mnaser: https://github.com/TecharoHQ/anubis/commit/1d700a037087a2a8695d9f7916f356be15f0a8e3#diff-36d3505065293db7ad1cfadc666dd4e150944a2d66dcb35acc6bca45ad8512dcL107 | 00:41 |
| @clarkb:matrix.org | that checks if the weight is >=25 | 00:41 |
| @clarkb:matrix.org | your weights were all 30. I'm guessing most of us are skimming under that threshold but your particular set of user agents are not | 00:41 |
| @clarkb:matrix.org | most of us == those of us not having the problem. The weight is an aggregate score accumulating a number of factors aiui | 00:42 |
| @clarkb:matrix.org | so ya maybe the correct thing for now is to disable the honeypot | 00:42 |
| @mnaser:matrix.org | either that or the git allow maybe? but you know best :) | 00:42 |
| @clarkb:matrix.org | the git allow is probably also good. But won't cover curl for example | 00:44 |
| @mnaser:matrix.org | ah correct, in my case i dont care, curl was my test vector, git is what i care about (pre-commit) but your call | 00:44 |
| @clarkb:matrix.org | yup I think its good belts and suspenders but to fix the general issue for now I think we disable the honeypot. I'll work on changes that do both | 00:45 |
| @clarkb:matrix.org | actually maybe one change to reduce the total number of restarts involved | 00:45 |
| -@gerrit:opendev.org- Clark Boylan proposed: [opendev/system-config] 999912: Disable the anubis honeypot https://review.opendev.org/c/opendev/system-config/+/999912 | 00:54 | |
| @clarkb:matrix.org | ok that bundles up a few of these things to avoid unnecessary restarts | 00:54 |
| @clarkb:matrix.org | I only modified the anubis config for gitea. I suspect that it is actually less of a problem for lists and wiki as they don't have an haproxy and don't use git clients | 00:55 |
| @clarkb:matrix.org | I need to eat dinner now. Hopefully CI comes back happy on that and we can land it over night or something | 00:55 |
| @mnaser:matrix.org | thank you for troubleshooting! | 00:56 |
| @clarkb:matrix.org | and thank you for helping to debug. The ip address and additional info was helpful to track downt he likely cause | 00:56 |
| @mnaser:matrix.org | np | 00:56 |
| @tkajinam:matrix.org | I'm wondering if anyone has seen this | 01:01 |
| ``` | ||
| $ git clone https://opendev.org/openstack/heat | ||
| Cloning into 'heat'... | ||
| fatal: https://opendev.org/openstack/heat/info/refs not valid: could not determine hash algorithm; is this a git repository? | ||
| $ git clone https://github.com/openstack/heat | ||
| Cloning into 'heat'... | ||
| remote: Enumerating objects: 146335, done. | ||
| ... | ||
| @mnaser:matrix.org | lol, yep. fix in flight - https://review.opendev.org/c/opendev/system-config/+/999912 | 01:02 |
| @mnaser:matrix.org | see chat log above ^ :) | 01:02 |
| @clarkb:matrix.org | its the discussion just above your message | 01:02 |
| @tkajinam:matrix.org | oops sorry it now loads | 01:02 |
| @clarkb:matrix.org | I really need to pop out now though. I think if that proposed change does not work then the next step is reverting back to 1.25.0 | 01:03 |
| @clarkb:matrix.org | infra-root ^ fyi for whoever is able to pick this up next | 01:03 |
| @mnaser:matrix.org | ouch fyi i just noticed ci jobs upstream re failing this too | 01:52 |
| @abelur:matrix.org | Morning all, could someone take a look at approving https://review.opendev.org/c/opendev/system-config/+/983318 (haproxy | 02:36 |
| PROXY protocol with gitea)? it's Clark's change I picked up. PS11 is green with a +2 from Michal, but no workflow+1 - I think the +2 landed | ||
| just before zuul reported verified. TIA | ||
| @abelur:matrix.org | Clark: - I've replied on 999195 with a sketch for converting iptables to nftables gradually instead of all at once (new inventory group, | 02:38 |
| opt-in per tier, ~15 changes). would appreciate a sanity check on the approach before I respin and continue on the approach. Appreciate if you can look into this whenever possible. | ||
| @abelur:matrix.org | * Clark: - I've replied on 999195 with a sketch for converting iptables to nftables gradually instead of all at once (new inventory group, opt-in per tier, ~15 changes). would appreciate a sanity check on the approach before I respin and continue on the approach whenever possible. | 02:42 |
| @harbott.osism.tech:regio.chat | FYI I've approved https://review.opendev.org/c/opendev/system-config/+/999912 now and will watch when it deploys | 06:07 |
| @harbott.osism.tech:regio.chat | for 983318 I think it would be better if clarkb has another look at it first | 06:23 |
| -@gerrit:opendev.org- Zuul merged on behalf of Dr. Jens Harbott: [openstack/project-config] 995781: Fix config errors for zuul-base-jobs https://review.opendev.org/c/openstack/project-config/+/995781 | 06:36 | |
| @harbott.osism.tech:regio.chat | hmm, the gate job seems stuck for more than half an hour according to the console log :( | 07:30 |
| @harbott.osism.tech:regio.chat | looks like the anubis container doesn't feel well. the docker log doesn't show anything obvious to me, though. trying to debug further | 07:35 |
| ``` | ||
| 170eb962a76c ghcr.io/techarohq/anubis:v1.26.2 "/ko-app/anubis" 42 minutes ago Up 42 minutes (unhealthy) gitea-docker_anubis_1 | ||
| ``` | ||
| -@gerrit:opendev.org- Zuul merged on behalf of Clark Boylan: [opendev/system-config] 999912: Disable the anubis honeypot https://review.opendev.org/c/opendev/system-config/+/999912 | 07:48 | |
| @harbott.osism.tech:regio.chat | seems it passed anyway, well then, let's see how the deployment goes. the issue sure affects a lot of our CI, too | 08:07 |
| @harbott.osism.tech:regio.chat | looks like the bad status was there before, like currently on gitea10: `Up 14 hours (unhealthy)` | 08:12 |
| @harbott.osism.tech:regio.chat | it looks like the change deployed, but didn't trigger any restarts. I'm not sure on the procedure for that, it there a script to go over all giteas and take them out of rotation? | 08:17 |
| @gthiemonge:matrix.org | looks like you're already working on issues with anubis, i wanted to report a few issues related to anubis blocking pip | 08:49 |
| @harbott.osism.tech:regio.chat | I did docker compose up -d on gitea10, will need to afk for a bit now | 08:58 |
| @harbott.osism.tech:regio.chat | I'm doing only `docker-compose up -d anubis` on the other servers now trying to minimize the impact, but it also seems like this is urgent affecting a lot of ppl | 09:35 |
| @harbott.osism.tech:regio.chat | hmm, looks like the problem is not yet fixed, like https://zuul.opendev.org/t/openstack/build/1c0ac6576f4a4074a870f5cbfd2c5457 still seems to be hitting some issue (not 100% sure whether the same thing, though) | 10:07 |
| @owenjones:matrix.org | Also still hitting it with git pulls | 10:08 |
| @harbott.osism.tech:regio.chat | yes, this one is more explicit, too https://zuul.opendev.org/t/openstack/build/0cc5a680f1864fe19a68c7a278cb6e4e | 10:10 |
| @harbott.osism.tech:regio.chat | I wonder whether we should revert the anubis bump until we can get a better grip on this | 10:24 |
| @harbott.osism.tech:regio.chat | it would also be interesting if someone can confirm whether the container was already unhealthy before the bump | 10:25 |
| @mnasiadka:matrix.org | Jens Harbott: I think https://review.opendev.org/c/opendev/system-config/+/983318 could help - we end up having LB address as x_real_ip and that's causing issues | 12:22 |
| @mnasiadka:matrix.org | Or maybe it would only improve our logging | 12:27 |
| -@gerrit:opendev.org- Michal Nasiadka proposed: [opendev/system-config] 999965: gitea: Fix healthcheck port https://review.opendev.org/c/opendev/system-config/+/999965 | 12:38 | |
| @mnasiadka:matrix.org | ^^ that will fix the healthchecks | 12:39 |
| @fungicide:matrix.org | mnasiadka: 999912 moved it into a metrics.bind setting in botPolicy.yaml, are you saying that won't work? do we need to remove it again there too, or do we need it set in both places? | 14:03 |
| @mnasiadka:matrix.org | fungi: anubis —healthcheck does not read config, just env var - don’t ask me why… | 14:07 |
| @fungicide:matrix.org | oh gotcha. i was just wondering if there's any point in keeping it set in botPolicy.yaml in that case, but i guess it's worth a shot | 14:09 |
| @mnasiadka:matrix.org | Well, default was 9090 or something like that, it’s listening mow on 15000 because of yaml, but not using it for checks | 14:15 |
| @fungicide:matrix.org | i see, so the env setting was used both by anubis service to decide what port to listen on and by the check cli to decide what port to connect to, but the service optionally also uses the bot policy to decide its listening port? | 14:20 |
| @clarkb:matrix.org | The docs say the env option is deprecated and you should use the yaml which is why I moved it. But I guess the check command hasn't been updated yet | 14:43 |
| @clarkb:matrix.org | Is the issue with blocking requests from UAs without explicit challenge rules still happening? | 14:44 |
| @clarkb:matrix.org | As a side note I do not consider it to be critical to fix this because our own CI jobs are affected. Those should be using zuul cached git checkouts instead. But clearly fixing this for everyone else is a good idea | 14:45 |
| @clarkb:matrix.org | But the updated ruleset explicitly allows git so is that not fixed with the change? | 14:45 |
| @clarkb:matrix.org | > <@harbott.osism.tech:regio.chat> it would also be interesting if someone can confirm whether the container was already unhealthy before the bump | 14:47 |
| No I think 1.26 broke the interpretation of the existing metric bind config so it tried to connect to http://localhost[::1]:15000 which is invalid. I think it worked on 1.25 | ||
| @clarkb:matrix.org | Is it the case that git is working now, but other non js capable clients like pip and requests etc are still failing? | 14:51 |
| @harbott.osism.tech:regio.chat | to me this looks like a git binary invocation that is still failing: https://zuul.opendev.org/t/openstack/build/0cc5a680f1864fe19a68c7a278cb6e4e | 14:56 |
| ``` | ||
| [ERROR]: Failed to clone a Git repository from `https://opendev.org/openstack/ansible-collection-kolla`: Command '['/usr/bin/git', 'clone', 'https://opendev.org/openstack/ansible-collection-kolla', '/home/zuul/.ansible/tmp/ansible-local-3414lmhi5_8t/tmpse98o48q/ansible-collection-kollacepngx6g']' returned non-zero exit status 128. | ||
| ``` | ||
| @clarkb:matrix.org | Ok so it's like we're back to wondering if our custom config is being completely ignored and we're using the default | 14:59 |
| @clarkb:matrix.org | Apparently if you start anubis with debug level logging it may tell you about config problems. Maybe we need to do that to see if it is falling back for some reason | 15:00 |
| @clarkb:matrix.org | I've pulled gitea09 out of the load balancer and will restart anubis there with debug logging and the metrics bind fix | 15:10 |
| @clarkb:matrix.org | `"loading policy file","fname":"/data/cfg/botPolicy.yaml"` | 15:11 |
| @fungicide:matrix.org | that's what we set as $POLICY_FNAME in the env and map as a volume into the container, at least | 15:13 |
| @clarkb:matrix.org | yup and our config is very straightforward and should explicitly allow git clients | 15:14 |
| @clarkb:matrix.org | but that is why we zeroed in on the honeypot last night because it appears to have weight based handling for things regardless of the bot config | 15:15 |
| @clarkb:matrix.org | but disabling the honeypot hasn't made things better | 15:15 |
| @clarkb:matrix.org | I'm going to put gitea09 in the rotation now and see if debug logging helps with requests coming in | 15:16 |
| @clarkb:matrix.org | with debug logging I can see it say `"check_result":{"name":"default/allow","rule":"ALLOW"` for git clients which matches my understanding of our ruleset | 15:20 |
| @gtema:matrix.org | are you guys now playing with anubis configuration? I get random failures in jobs with cryptic errors during pip installing packages from opendev: "raise InvalidMarker(\npip._vendor.packaging.markers.InvalidMarker: Invalid marker: 're not a bot!</title><link rel=\"stylesheet\" href=\"/.within.website/x/xess/xess.min.css?cachebuster=v1.26.2\"><meta name=\"viewport\" content=\"width=device-width, initial-scale=1.0\"><meta name=\"robots\" content=\"noindex,nofollow\"><style>', parse error at 're not a'\n"," | 15:22 |
| @clarkb:matrix.org | part of me wonders if fixing the metrics bind config fixes config in geneal and we stop falling back | 15:22 |
| @fungicide:matrix.org | gtema: after an upgrade from 1.25 to 1.26 some behavior changed and we're trying to iron them out | 15:23 |
| @gtema:matrix.org | ok, thanks for explanation | 15:23 |
| @clarkb:matrix.org | gtema: yes, but that is due to the errors you're seeing not causing them | 15:27 |
| @clarkb:matrix.org | or at least not causing them any worse. We upgrade anubis yesterday and it appears to have started acting in ways that are not aligned iwth the expected config | 15:27 |
| @clarkb:matrix.org | we're trying to sort that out | 15:27 |
| @gtema:matrix.org | good, looking forward, since landing changes in some repos is now literally impossible. Thanks for taking care | 15:28 |
| @fungicide:matrix.org | just a reminder that you can configure zuul to provide you with git checkouts, and that will even make depends-on between repositories viable. doing a pip install of a repository over the network is not as efficient | 15:29 |
| @clarkb:matrix.org | right this shouldn't be preventing you from merging any code within our systems | 15:30 |
| @clarkb:matrix.org | its definitely a problem but if it is causing that problem your jobs are argbuably buggy | 15:31 |
| @gtema:matrix.org | and even the pep8 job is failing which is using pre-commit to install hacking, but that is a completely different style | 15:32 |
| @clarkb:matrix.org | yes and I've said for tyaers that pre commit should be deleted for this very reason | 15:32 |
| @gtema:matrix.org | fungi, I even do it this way, but it still fails every time on different place | 15:32 |
| @clarkb:matrix.org | its entire installation system is bad/broken/confusing/terrible | 15:33 |
| @clarkb:matrix.org | and completely undermines CI systems like ours | 15:33 |
| @fungicide:matrix.org | shouldn't precommit should be installing releases of hacking from pypi, not via git? | 15:34 |
| @gtema:matrix.org | amen - I am extraordinary frustrated myself from the setup we have in some areas and how much unnecessary effort is required to do a simple thing | 15:34 |
| @gtema:matrix.org | fungi: at least in keystone it is now: ``` - repo: https://opendev.org/openstack/hacking``` | 15:34 |
| @clarkb:matrix.org | looks like gitea14 was not updated overnight | 15:35 |
| @fungicide:matrix.org | i wonder why it was configured to do that | 15:35 |
| @clarkb:matrix.org | anubis is still 21 hours old there | 15:35 |
| @clarkb:matrix.org | fungi: gtema beacuse pre commit does not know how to install from pip without jumping through a bunch of undocumented hoops | 15:35 |
| @clarkb:matrix.org | its the design of the tool | 15:35 |
| @clarkb:matrix.org | so now I'm wondering if the errors we've seen overnight are due to clients hitting gitea14 | 15:35 |
| @clarkb:matrix.org | I'm going to fix the metrics by hand there and restart anubis on gitea14. At that point metrics and config will be updated/fixed on all the backends | 15:36 |
| @fungicide:matrix.org | oh, because we rolled out a newer configuration but anubis didn't get restarted? | 15:36 |
| @clarkb:matrix.org | so far in monitoring logs after doing that I haven't seen any obvious errors from git/curl/pip etc `tail -f /var/log/containers/docker-anubis.log | grep -v Mozilla` | 15:36 |
| @clarkb:matrix.org | fungi: yes Jens Harbott manually restarted things but maybe missed this one | 15:36 |
| @clarkb:matrix.org | That is done. I think my operating assumption now is that our updates (possibly includign the fix for the metrics bind) do improve things, but due to incomplete rollout we were still getting some errors | 15:38 |
| @clarkb:matrix.org | we should monitor now for any new errors after 15:38UTC | 15:38 |
| @fungicide:matrix.org | who has/had a viable reproducer for pip install or git fetch failures? can we get them to retry and confirm it's clear now? | 15:38 |
| @gtema:matrix.org | I started recheck in one of the changes (https://review.opendev.org/c/openstack/codegenerator/+/999979) that was struggling a lot | 15:40 |
| @clarkb:matrix.org | load balancer reports all 6 up for http and https | 15:40 |
| @fungicide:matrix.org | fwiw, postorius/hyperkitty on the lists01 sites seem to be working fine with newer anubis so far | 15:46 |
| @fungicide:matrix.org | but it's also not behind haproxy or anything complicated | 15:47 |
| @clarkb:matrix.org | yes I think it is still likely that this is an annoying interaction between being behind the single IP load balancer and the honeypot | 15:47 |
| @clarkb:matrix.org | those two don't have the load balancer so the honeypot is probably fine for them | 15:47 |
| -@gerrit:opendev.org- Roja Eswaran proposed: [openstack/diskimage-builder] 999989: debootstrap: add DIB_MMDEBSTRAP_EXTRA_ARGS support https://review.opendev.org/c/openstack/diskimage-builder/+/999989 | 15:49 | |
| @mnasiadka:matrix.org | Clark: shouldn't we think about merging the PROXY protocol patch? It would at least give us more visibility into the real ip and we could reenable honeypot? | 16:01 |
| @fungicide:matrix.org | yes, i think that's next in line once we have the other config change deployed | 16:03 |
| @clarkb:matrix.org | I want to recheck it and look at its logs as soon as the metrics fix is in. This way we can confirm (or not) that anubis 1.26 with the running config sees the upstream client IP properly | 16:04 |
| @clarkb:matrix.org | we should have a change to fix the metrics/healthcheck for lists too | 16:06 |
| -@gerrit:opendev.org- Zuul merged on behalf of Michal Nasiadka: [opendev/system-config] 999965: gitea: Fix healthcheck port https://review.opendev.org/c/opendev/system-config/+/999965 | 16:07 | |
| @clarkb:matrix.org | https://review.opendev.org/c/opendev/system-config/+/983318 has been rechecked | 16:08 |
| @clarkb:matrix.org | upstream gerrit confirms that we should create stable branches for the zuul-results-summary plugin and suggests I can do that myself somehow. So I'll look at that I guess | 16:09 |
| @mnasiadka:matrix.org | Nice that Anubis has a metrics endpoint, we can scrape it with Prometheus in the future | 16:10 |
| @clarkb:matrix.org | https://gerrit.googlesource.com/plugins/zuul-results-summary/ stable-3.13 and stable-3.14 exist now. I'll push an update to our gerrit 3.13.8 upgrade change now | 16:13 |
| -@gerrit:opendev.org- Clark Boylan proposed: [opendev/system-config] 994938: Update Gerrit images to 3.13.8 and 3.14.2 https://review.opendev.org/c/opendev/system-config/+/994938 | 16:16 | |
| @clarkb:matrix.org | based on my skimming of zuul job failures and anubis log greps I think things are looking happier so far | 16:19 |
| @clarkb:matrix.org | I see two denials due to requests coming from cloudflare workers that are expected. I don't see any other non browser requets getting logged/challenged yet | 16:19 |
| @gtema:matrix.org | I can confirm, my change which was heavily failing above is happy | 16:19 |
| @clarkb:matrix.org | pip is complaining about short reads from the mirrors (which should be unrelated no anubis involved there) | 16:22 |
| @fungicide:matrix.org | yes, our "mirrors" (caching proxies) for pypi aren't using anubis in any way | 16:23 |
| @mnasiadka:matrix.org | During the day I had multiple cases of some weird Anubis loop, where it wanted me to attest I'm a human for every URL I was going through one by one (and once it didn't even show a progress bar) - but hopefully with the PROXY change we can easier debug such problems | 16:23 |
| @fungicide:matrix.org | i've gotten the missing progress bar before too, i think it's an incomplete load or the browser fails to connect back to fetch something | 16:24 |
| @fungicide:matrix.org | a refresh has always fixed it for me when i do hit that | 16:24 |
| @clarkb:matrix.org | and unclear if that is an upstream pypi problem or something on our end. Not sure I'll dig into that right now given the other stuff I'm looking at | 16:25 |
| @clarkb:matrix.org | the anubis situation continues to look stable and happy. I'm going to pop out now and do morning things that I skipped earlier. I'll followup on the PROXY protocol change once its recheck is done and make sure that gerrit image builds look happy after my stable branch creation and update to the change to use the stable branch for the plugin. | 16:41 |
| @fungicide:matrix.org | thanks! | 16:42 |
| @clarkb:matrix.org | somneone else may want to look at the pip short reads if they persist | 16:43 |
| @clarkb:matrix.org | https://status.python.org/ indicates things are operating normally but does show some spikes in CDN file error rate (maybe related?) | 16:44 |
| @mnaser:matrix.org | yes just another note for the stuck / missing progresss bar fixed by a refresh here :) | 16:44 |
| @clarkb:matrix.org | I haev a meeting but I notice that the pip short reads look like they may be coming from our whl cache and not the pypi mirror | 17:59 |
| @clarkb:matrix.org | for example https://zuul.opendev.org/t/openstack/build/546b5bd404f7477c94ffd949a74d8448 appears to say it is downloading a docker wheel which does not exist on pypi: https://pypi.org/project/docker/6.1.3/#files | 17:59 |
| @clarkb:matrix.org | so maybe there is a problem with afs in general or with a specific mirror node? | 17:59 |
| @clarkb:matrix.org | `[Thu Aug 6 13:45:41 2026] afs: disk cache read error in CacheItems slot ......` and `[Thu Aug 6 13:45:41 2026] afs: Error while alloc'ing cache slot for file ....` so ya something with that mirror I think | 18:03 |
| @clarkb:matrix.org | that is from dmesg -T | 18:03 |
| @fungicide:matrix.org | which server? | 18:05 |
| @clarkb:matrix.org | * that is from dmesg -T on mirror.dfw.rax | 18:05 |
| @clarkb:matrix.org | sorry I've been having matrix issues with half my messages not going thourhg. I edited my previous message to indicate that was mirror.dfw.rax which is where my example link aboev failed | 18:06 |
| @fungicide:matrix.org | oh neat, i just happened to have picked that one to look at dmesg on too | 18:07 |
| @fungicide:matrix.org | so looks like it might be a local problem with the cache on that one server, i don't see similar errors on other mirrors | 18:10 |
| @fungicide:matrix.org | might be worth rebooting that server to see if the i/o error clears | 18:13 |
| @clarkb:matrix.org | should we disable the region in opendev/zuul-providers first? | 18:14 |
| @fungicide:matrix.org | though there weren't similar errors for `/dev/xvdb1` which is the underlying block device serving as a pv | 18:14 |
| @clarkb:matrix.org | a reboot will definitely be noticed by jobs | 18:14 |
| @clarkb:matrix.org | whereas its unclear if this is a 100% failure state right now with the cache errors | 18:14 |
| @fungicide:matrix.org | yeah, we can pause this region. does zuul-launcher have that option via cli/rpc or was that only in nodepool? | 18:15 |
| @clarkb:matrix.org | I think I've only ever done it via config updates | 18:15 |
| @clarkb:matrix.org | which apply immediately now unlike nodepool which needed to have a deployment pass | 18:16 |
| @fungicide:matrix.org | that's a good point | 18:16 |
| @fungicide:matrix.org | just set `instances: 0` for now, or is there a pause feature for the config we haven't used yet? not finding one in zuul docs if it exists | 18:19 |
| @clarkb:matrix.org | yes just set the limit to 0. Ie60d558cc1ee9bfd0f18b54ed8fd32053a014aa4 is an example | 18:19 |
| -@gerrit:opendev.org- Jeremy Stanley https://matrix.to/#/@fungicide:matrix.org proposed: [opendev/zuul-providers] 1000022: Temporarily disable rax-dfw https://review.opendev.org/c/opendev/zuul-providers/+/1000022 | 18:20 | |
| -@gerrit:opendev.org- Zuul merged on behalf of Jeremy Stanley https://matrix.to/#/@fungicide:matrix.org: [opendev/zuul-providers] 1000022: Temporarily disable rax-dfw https://review.opendev.org/c/opendev/zuul-providers/+/1000022 | 18:25 | |
| @clarkb:matrix.org | looking at https://zuul.opendev.org/t/openstack/build/ea9e6228e92c4cf6a05e5e68d24d8cb8/log/gitea99.opendev.org/containers/docker-anubis.log I think the real ip there is properly being recorded with PROXY protocl in place (the first set of requests are from some random crawler on the internet and the last few are from our tests which originate from 127.0.0.1) | 18:33 |
| @fungicide:matrix.org | so apache gets it from haproxy via proxy protocol (outside tls encryption), then apache terminates the encryption and conveys the information via x-forwarded-for to anubis i guess? | 18:37 |
| @clarkb:matrix.org | yes I think so | 18:48 |
| @clarkb:matrix.org | ok I +1'd https://review.opendev.org/c/opendev/system-config/+/983318 and didn't +2 it because I wrote some of the original change. Though at this point Anil Belur figured most of it out. I did leave a comment about some deployment impact concerns we should be aware of for others who review it | 18:54 |
| @clarkb:matrix.org | I think the gerrit image builds are working now too | 18:54 |
| @clarkb:matrix.org | so I think fixing rax dfw's mirror is item 0, then we can consider if we want to proceed with gerrit upgrades or deploying proxy protocol support to gitea's load balancer setup. Then I haev a bunch of changes I've promised people I would review so need to go look at those after lunch | 19:00 |
| @clarkb:matrix.org | fungi: were you going to reboot mirror.dfw.rax? | 19:09 |
| @clarkb:matrix.org | ok lunch now. I'll check back in a bit | 19:11 |
| @fungicide:matrix.org | i was going to wait until jobs there had trailed off... checking now | 19:12 |
| @clarkb:matrix.org | oh ya I guess we could still have jobs running on nodes from there | 19:12 |
| @clarkb:matrix.org | its only the new launches we stopped | 19:12 |
| @fungicide:matrix.org | https://grafana.opendev.org/d/fd44466e7f/zuul-launcher3a-rackspace?orgId=1&from=now-24h&to=now&timezone=utc&var-region=$__all shows it trailing off | 19:13 |
| @fungicide:matrix.org | down to 17 | 19:14 |
| @fungicide:matrix.org | rather 14 | 19:14 |
| @fungicide:matrix.org | they're probably mostly past the point where they would be fetching things from the mirror, but i'll give it a little longer | 19:14 |
| @fungicide:matrix.org | corvus: i don't suppose you recall what led to our gerrit change #1 being created about 10 months after change 1000? | 19:46 |
| @fungicide:matrix.org | some brief configuration error? gerrit bug? | 19:46 |
| @fungicide:matrix.org | oh! the date in gerrit is wrong | 19:47 |
| @fungicide:matrix.org | the actual git commit was created 2011-07-18 | 19:47 |
| @fungicide:matrix.org | grafana says we're down to 3 nodes in use for rax-dfw so i'll go ahead and reboot the mirror server there now | 20:00 |
| @fungicide:matrix.org | unrelated, looks like the gerrit upgrade change is working again | 20:02 |
| @fungicide:matrix.org | okay, mirror03.dfw.rax is online again if anyone wants to retry the previously failing path | 20:03 |
| -@gerrit:opendev.org- Jeremy Stanley https://matrix.to/#/@fungicide:matrix.org proposed: [opendev/zuul-providers] 1000037: Revert "Temporarily disable rax-dfw" https://review.opendev.org/c/opendev/zuul-providers/+/1000037 | 20:04 | |
| @clarkb:matrix.org | fungi: I'm not sure we had an easy reproducer because pip doesn't actually tell you what specific request short read | 20:06 |
| @clarkb:matrix.org | but I was able to download a wheel from the mirror cache just now and I think it worked | 20:06 |
| @clarkb:matrix.org | I +2'd the restoration change and if we can't come up with a better test I guess we let zuul find out | 20:07 |
| @fungicide:matrix.org | wfm, thanks! | 20:07 |
| @clarkb:matrix.org | yup I created stable-3.13 and stable-3.14 branches on the plugin after davido confirmed that was the correct solution. Then updated the change to pull those versions in our builds | 20:09 |
| @clarkb:matrix.org | fungi: should I go ahead and approve 1000037 then? | 20:13 |
| @fungicide:matrix.org | yeah, sounds good to me | 20:21 |
| @clarkb:matrix.org | Anil Belur: responded on the nftables change. I think your plan sounds good to me and I tried to answer your questions. Let me know if there are other questions or concerns | 20:22 |
| @fungicide:matrix.org | i'm going to have to adjust my gertty config to accommodate an extra digit in the change number column now | 20:22 |
| -@gerrit:opendev.org- Zuul merged on behalf of Jeremy Stanley https://matrix.to/#/@fungicide:matrix.org: [opendev/zuul-providers] 1000037: Revert "Temporarily disable rax-dfw" https://review.opendev.org/c/opendev/zuul-providers/+/1000037 | 20:22 | |
| @clarkb:matrix.org | fwiw I continue to see happy anubis logs | 20:25 |
| @fungicide:matrix.org | i ended up shimming anubis onto one of my aging servers that hosts some gitweb, a moinmoin wiki, roundcube webmail... | 20:28 |
| @fungicide:matrix.org | though i went with v1.27.0-pre3 since it's not critical | 20:29 |
| @fungicide:matrix.org | worked pretty well, the main catch was i had to fiddle with uwsgi processes that were expecting to see they were served via https and weren't any longer due to the additional vhost layer/proxy | 20:30 |
| @clarkb:matrix.org | fungi: https://zuul.opendev.org/t/openstack/build/d52be960b0b34ee1ab20bc5cf0bebb3f I don't think the reboot fixed the mirror | 20:38 |
| @clarkb:matrix.org | however dmesg doesn't report new cache errors | 20:38 |
| @fungicide:matrix.org | possible those were unrelated | 20:39 |
| @fungicide:matrix.org | though is it only failing in rax-dfw? | 20:39 |
| @clarkb:matrix.org | `AH00052: child pid 1890 exit signal Segmentation fault (11)` | 20:39 |
| @clarkb:matrix.org | yes the problem is specific to that mirror | 20:40 |
| @fungicide:matrix.org | maybe we need to flush the cache there somehow | 20:40 |
| @clarkb:matrix.org | `(28)No space left on device:` | 20:40 |
| @fungicide:matrix.org | d'oh! | 20:40 |
| @clarkb:matrix.org | I think the issue is the apache cache not the openafs cache | 20:40 |
| @clarkb:matrix.org | its full (so pruning isn't going quickly enough?) and that leads to segfaults maybe when the apache process tries to write and it cannot? | 20:41 |
| @fungicide:matrix.org | /dev/mapper/main-apache2 126G 126G 16K 100% /var/cache/apache2 | 20:41 |
| @clarkb:matrix.org | the htcacheclean runs hourly starting at X:00 | 20:42 |
| @fungicide:matrix.org | yeah, it's been going for about 40 minutes | 20:42 |
| @clarkb:matrix.org | so maybe we take the at server back out of service and see how much disk it frees when it run in ~18 minutes? | 20:42 |
| @clarkb:matrix.org | fungi: no thats a different one | 20:42 |
| @clarkb:matrix.org | that one is for the default cache not the one we set up for mirror caching | 20:42 |
| @fungicide:matrix.org | ah! | 20:42 |
| @clarkb:matrix.org | the one for mirror caching can be seen via crontab -l and isn't running yet | 20:42 |
| @fungicide:matrix.org | yeah i guess i rebooted it right at the top of the hour and that got missed | 20:43 |
| @fungicide:matrix.org | so it's probably an hour behind now | 20:43 |
| @clarkb:matrix.org | ya though this seems likely the original issue too | 20:44 |
| @fungicide:matrix.org | quite likely, yes | 20:44 |
| @clarkb:matrix.org | since the behavior hasn't changed on the client side. But could be two different problems i suppose | 20:44 |
| @fungicide:matrix.org | i can manuallt start a more aggressive htcacheclean in a root screen session | 20:44 |
| @clarkb:matrix.org | ++ | 20:44 |
| @clarkb:matrix.org | do you want me to propose a change ot take the region back out of service? or leave it in in hopes the cleanup gives us room? | 20:45 |
| @fungicide:matrix.org | i'm hoping starting htcacheclean now frees up enough to get it to stop erroring | 20:45 |
| @clarkb:matrix.org | ack | 20:45 |
| @fungicide:matrix.org | i have it running set to a target of 30000M instead of 60000M | 20:46 |
| @clarkb:matrix.org | we're up to 60k of free disk now too :P | 20:46 |
| @clarkb:matrix.org | we might be too far behind for this to catch up while things are running seems to bounce around between 8k and 100k or so free | 20:47 |
| @clarkb:matrix.org | I'll give it a few minutes. If we're still under 1M free space I'll push a change to disable the region again | 20:50 |
| @fungicide:matrix.org | yeah, i'm good with that | 20:51 |
| @fungicide:matrix.org | once htcacheclean falls behind, it becomes sisyphean | 20:52 |
| -@gerrit:opendev.org- Clark Boylan proposed: [opendev/zuul-providers] 1000040: Disable rax-dfw again due to continued mirror issues https://review.opendev.org/c/opendev/zuul-providers/+/1000040 | 20:55 | |
| @clarkb:matrix.org | 52k at last check so I pushed ^ | 20:55 |
| @clarkb:matrix.org | fungi: do you want to approve that or should I just hit the button? | 20:55 |
| @fungicide:matrix.org | done | 20:55 |
| -@gerrit:opendev.org- Zuul merged on behalf of Clark Boylan: [opendev/zuul-providers] 1000040: Disable rax-dfw again due to continued mirror issues https://review.opendev.org/c/opendev/zuul-providers/+/1000040 | 20:56 | |
| @clarkb:matrix.org | Eric Ball: left a note on https://review.opendev.org/c/opendev/system-config/+/996553 I think it would be good to split that change up into multiple changes so that we don't update everything all at once and potentially have a bunch of broken services at the same time | 21:04 |
| @clarkb:matrix.org | my comment on that change has a suggested break down | 21:05 |
| @clarkb:matrix.org | fungi: we have 88GB free on that server now and I think your htcacheclean run completed. Should we rerun it one more time to see if it is near steady state? | 21:07 |
| @clarkb:matrix.org | then we can probably reenable the server | 21:07 |
| @fungicide:matrix.org | yeah, it looks done | 21:07 |
| @fungicide:matrix.org | i'm going to rerun it again quickly though | 21:07 |
| @fungicide:matrix.org | it was running while things were still being added | 21:08 |
| @fungicide:matrix.org | once this run gets it down to 30gb of cached data we can merge the revert | 21:08 |
| @fungicide:matrix.org | and then keep an eye on it for a bit to see how fast it grows | 21:09 |
| @clarkb:matrix.org | ++ | 21:09 |
| @fungicide:matrix.org | and in theory the 22z cronjob will knock it back down if it gets above 60gb | 21:09 |
| @fungicide:matrix.org | okay it completed and got utilization down to 34gb according to `df -h` which i guess is as good as it gets | 21:13 |
| @fungicide:matrix.org | is there a revert up already? | 21:13 |
| @clarkb:matrix.org | no revert yet | 21:15 |
| -@gerrit:opendev.org- Clark Boylan proposed: [opendev/zuul-providers] 1000044: Revert "Disable rax-dfw again due to continued mirror issues" https://review.opendev.org/c/opendev/zuul-providers/+/1000044 | 21:15 | |
| @clarkb:matrix.org | there just pushed one | 21:15 |
| -@gerrit:opendev.org- Zuul merged on behalf of Clark Boylan: [opendev/zuul-providers] 1000044: Revert "Disable rax-dfw again due to continued mirror issues" https://review.opendev.org/c/opendev/zuul-providers/+/1000044 | 21:18 | |
| @clarkb:matrix.org | up to 52GB disk used | 21:39 |
| @fungicide:matrix.org | yeah, i've got `watch -n60 df -h /var/cache/apache2` running in a terminal | 21:41 |
| @fungicide:matrix.org | curious to see if it self-corrects at the top of the hour or starts to get behind again | 21:41 |
| @clarkb:matrix.org | looks like cache cleaning should start with about 79Gb of cache disk used | 21:59 |
| @clarkb:matrix.org | maybe even 80GB | 21:59 |
| @fungicide:matrix.org | yeah, it's up to 82 now | 22:01 |
| @clarkb:matrix.org | I think htcacheclean does a data gathering sweep then a cleanup sweep though so probably can't say much about its ability to keep up until its done? | 22:02 |
| @fungicide:matrix.org | right | 22:02 |
| @fungicide:matrix.org | already up to 84gb used | 22:02 |
| @clarkb:matrix.org | looks like it is done running and we're at 69GB | 22:09 |
| @clarkb:matrix.org | I wonder how close to the limit that puts us at 2300 UTC | 22:10 |
| @clarkb:matrix.org | it is possible we may need to run the cleanup more often? | 22:10 |
| @fungicide:matrix.org | maybe. we're up to 75gb used again already | 22:14 |
| @fungicide:matrix.org | on a positive note, it seems like it's probably an accumulation of a few large files rather than numerous small ones, so htcacheclean is completing relatively quickly | 22:14 |
| @fungicide:matrix.org | the real problem is when we get into situations where it takes htcacheclean so long to complete that it can never catch up | 22:15 |
| @clarkb:matrix.org | fungi: looks like it is full right now | 22:55 |
| @clarkb:matrix.org | so it iddn't make it the hour | 22:55 |
| @clarkb:matrix.org | it is curious that other mirrors don't seem to hae the same issue. Maybe that is just chance/luck? | 22:56 |
| @clarkb:matrix.org | but I guess we should disable the region again then maybe run the cron job more aggressively? | 22:56 |
| -@gerrit:opendev.org- Clark Boylan proposed: [opendev/system-config] 1000055: Run apache cache cleanups more often https://review.opendev.org/c/opendev/system-config/+/1000055 | 22:59 | |
| @fungicide:matrix.org | it's probably somewhat relative to the region quota | 23:00 |
| -@gerrit:opendev.org- Clark Boylan proposed: [opendev/zuul-providers] 1000057: Reapply "Disable rax-dfw again due to continued mirror issues" https://review.opendev.org/c/opendev/zuul-providers/+/1000057 | 23:00 | |
| @clarkb:matrix.org | first change is the run it more often idea and second one disables the region until it is under control | 23:01 |
| @clarkb:matrix.org | fungi: I wonder too if someone is running a job that causes us to cache several 30GB objects too | 23:05 |
| @clarkb:matrix.org | anyway I will approve 1000057 shortly if I don't hear any other feedback | 23:05 |
| -@gerrit:opendev.org- Zuul merged on behalf of Clark Boylan: [opendev/zuul-providers] 1000057: Reapply "Disable rax-dfw again due to continued mirror issues" https://review.opendev.org/c/opendev/zuul-providers/+/1000057 | 23:07 | |
| @fungicide:matrix.org | i need to step away for the evening, but will check on this again first thing tomorrow before my meetings | 23:10 |
| @clarkb:matrix.org | thanks. I too am running out of steam after what felt like an early start today (it wasn't that early; I'll blame jetlag) | 23:12 |
Generated by irclog2html.py 4.1.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!