| @clarkb:matrix.org | fungi: I think we are, but if we can get as far as haproxy to apache that is one fewer hop to debug through until anubis adds support for the proxy protocol. I'm doing local system updates then have a change or two to push then will look at that update | 21:27 |
|---|---|---|
| @clarkb:matrix.org | the ideal is being able to trace it the whole way but being able to trace sub path segments is an improvement over what we have today with the manual tracgin and port numbers | 21:28 |
| -@gerrit:opendev.org- Clark Boylan proposed: [opendev/system-config] 998741: Update anubis to 1.26.0 https://review.opendev.org/c/opendev/system-config/+/998741 | 22:12 | |
| -@gerrit:opendev.org- Clark Boylan proposed: [opendev/system-config] 998742: Update Gitea to 1.27.0 https://review.opendev.org/c/opendev/system-config/+/998742 | 22:44 | |
| @clarkb:matrix.org | ok those are the two changes I wanted to prpose | 22:44 |
| @clarkb:matrix.org | Anil Belur: fungi I posted a comment to https://review.opendev.org/c/opendev/system-config/+/983318 I think there is a test specific issue (again with the use of private vs public IPs though due to apache not ansible). I think the ideal update would be to update the testinfra test cases to talk to our public IP address so that we don't need split configs in test and production. The one gotcha there is making it work with floating Ips but I think if we use the public ip address listed by zuul then it should just work? I'm not quite sure how to plumb that into testinfra test cases though | 23:00 |
| @abelur:matrix.org | Clark: Thanks for the review. I've dug into the AH03500 - it's the IPv4-mapped address, not the trust config. gitea-lb binds :::443 dual-stack, so the testinfra curl to 127.0.0.1 (and any v4 client) arrives as ::ffff:127.0.0.1 and haproxy send-proxy forwards PROXY TCP6 ::ffff:127.0.0.1 ... . Released Apache 2.4.x mod_remoteip validates a TCP6 address against charset 0123456789abcdefABCDEF: - no dot - so the mapped form fails strspn() → AH03500 → HDR_ERROR → connection aborted. Only unreleased httpd trunk added the . . | 23:46 |
| @abelur:matrix.org | So RemoteIPInternalProxy won't help - the parse aborts before any trusted/internal-proxy check runs; the LB is already trusted, it's the address string that's rejected. And pointing the tests at the public IPv4 won't help either: on the dual-stack socket it's still ::ffff:<public_v4> → same error. I also don't think it's test-specific - the same bind maps every real IPv4 visitor, so as-is this would abort IPv4 clients to opendev.org in prod once PROXY protocol is on. The failing tests are basically a canary. | 23:47 |
| @clarkb:matrix.org | Anil Belur: RemoteIPTrustedProxy doesn't trust client IPs that are on private network address ranges | 23:47 |
| @clarkb:matrix.org | even if you trust the proxy address | 23:48 |
| @abelur:matrix.org | the Cleanest fix looks haproxy-side: bind native sockets instead of one dual-stack - bind 0.0.0.0:443 + bind :::443 v6only. Then v4 clients get PROXY TCP4 ... (dots are legal in the TCP4 charset), v6 stays native TCP6. Fixes the tests and prod, no testinfra/public-IP plumbing, no test-vs-prod split. Have it staged locally. Any reason gitea-lb needs the single dual-stack :::443 ? | 23:48 |
| @clarkb:matrix.org | Anyway either way it sounds like a solution is to request the resources via the public address rather than 127.0.0.1 then we don't need to special case the 127.0.0.1? | 23:48 |
| @clarkb:matrix.org | oh any ipv4 mapped to ipv6 would fail as is I guess? | 23:49 |
| @clarkb:matrix.org | then ya lisening on both 0.0.0.0 and :: should fix it. That should also be self testing via those test cases | 23:49 |
| @clarkb:matrix.org | I wonder if we would then hit the RemoteIPInternalProxy thing? Maybe we need both? | 23:50 |
| @abelur:matrix.org | oh yeah - any v4-mapped-v6 fails as-is ... also worth noting the PROXY path never checks RemoteIPTrustedProxy/InternalProxy though; it sets the client IP straight from the header and dies earlier on the charset (TCP6 charset has no . , so ::ffff:127.0.0.1 -> AH03500 → abort). | 23:54 |
| @abelur:matrix.org | So "use the public address" only works for public IPv6 - public IPv4 still maps to ::ffff: and fails, and v4-only nodes have no v6. | 23:55 |
| @clarkb:matrix.org | got it. I'm more thinking ahead to fixing that problem with the separate ipv4 and ipv6 listen/binds if we'll hit the issue I thought we were hitting | 23:56 |
| @abelur:matrix.org | its cleaner to fix at the LB bind (bind 0.0.0.0:443 + bind :::443 v6only ) so v4 is native PROXY TCP4: no 127.0.0.1 special-casing, and it also saves real v4 users in prod ? | 23:56 |
| @clarkb:matrix.org | if not great, but if so then we can also update the test case to hit the public address to avoid the issue | 23:56 |
Generated by irclog2html.py 4.1.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!