| -@gerrit:opendev.org- Dr. Jens Harbott proposed: [opendev/zuul-providers] 1000965: Update cached images https://review.opendev.org/c/opendev/zuul-providers/+/1000965 | 12:55 | |
| @clarkb:matrix.org | as a heads up I have approved https://review.opendev.org/c/opendev/system-config/+/1000909 which should upgrade gitea in a little bit | 15:12 |
|---|---|---|
| @fungicide:matrix.org | thanks! i'm around | 15:14 |
| -@gerrit:opendev.org- Zuul merged on behalf of Jeremy Stanley https://matrix.to/#/@fungicide:matrix.org: [openstack/project-config] 1000868: Update openstack-github-mirroring API token https://review.opendev.org/c/openstack/project-config/+/1000868 | 15:17 | |
| @clarkb:matrix.org | fungi: if you can review https://review.opendev.org/c/opendev/system-config/+/983318 that would be great. This is the PROXY protocol change for gitea whcih I think we can consider applying after the gitea upgrade as well, but it could use another reviewer. I'm only +1 on it because I wrote some of the change and its a big enough update that I think having two people who didn't write the change weigh in is a good idea | 15:21 |
| @fungicide:matrix.org | wow, the commit message there has become a novella | 15:23 |
| @fungicide:matrix.org | lgtm | 15:25 |
| -@gerrit:opendev.org- Zuul merged on behalf of Clark Boylan: [opendev/system-config] 1000909: Upgrade gitea to 1.27.2 https://review.opendev.org/c/opendev/system-config/+/1000909 | 16:05 | |
| @clarkb:matrix.org | that deployment has gotten through gitea10 which is what I'm currently talking to and it seems to work in my browser | 16:14 |
| @clarkb:matrix.org | I'll make sure all of the backends get updated. Then fungi I guess we decide if we're proceeding with the PROXY change at that point? | 16:15 |
| @fungicide:matrix.org | yep | 16:15 |
| @fungicide:matrix.org | sounds good to me | 16:15 |
| @fungicide:matrix.org | i think we can if everything still looks okay | 16:15 |
| @fungicide:matrix.org | (once this deploy finishes i mean) | 16:15 |
| @clarkb:matrix.org | yes, probably a good idea | 16:15 |
| @clarkb:matrix.org | all of the backends are updated now | 16:21 |
| @clarkb:matrix.org | I can clone system-config successfully and the deploy buildset is marked a success. I think that all looks good. THe other major thing to check is replication | 16:22 |
| @fungicide:matrix.org | yeah, cloning and browsing repositories is working fine for me as well | 16:23 |
| @clarkb:matrix.org | I think https://review.opendev.org/c/starlingx/docs/+/979904 replicated to https://opendev.org/starlingx/docs/commit/09d253bd531483376f17ae6a946e4b2593666f11 after I noted all of the backends are updated above (its close though) | 16:23 |
| @clarkb:matrix.org | I think we can probably go ahead and approve the PROXY change if we're happy with ^ fungi I'm still looking for a newer patchset/change update ot check replication in a more claerly post upgrade timeframe | 16:27 |
| @fungicide:matrix.org | approved now | 16:27 |
| @clarkb:matrix.org | https://review.opendev.org/c/starlingx/config/+/999577 definitely replicated to https://opendev.org/starlingx/config/commit/3635c8b0f9f008ee286f301c4dee50329051f247 after the upgrade | 16:35 |
| -@gerrit:opendev.org- Zuul merged on behalf of Clark Boylan: [opendev/system-config] 983318: Use Haproxy PROXY Protocol with Gitea https://review.opendev.org/c/opendev/system-config/+/983318 | 17:36 | |
| @clarkb:matrix.org | that updated some ansible vars so its running a few extra jobs and will be a minute before it starts running the gitea specific job that should actually make changes | 17:39 |
| @clarkb:matrix.org | though I'm noticing now that it will deploy the gitea load balancer concurrently with the letsencrypt job? Anyway this is the big that I was semi worried about with load balacner config being out of sync with apache creating errors. but I think we can see where we end up and ride it out. It should only be a few minutes | 17:41 |
| @fungicide:matrix.org | watching | 17:44 |
| @clarkb:matrix.org | I have confirmed that opendev.org is no longer reachable and the haproxy log reports NOSRV | 17:47 |
| @clarkb:matrix.org | this is the outage I was worried about where we now need to update the backends to be in sync with the frontend | 17:47 |
| @clarkb:matrix.org | the gitea deployment has started so hopefully things are happy shortly | 17:48 |
| @fungicide:matrix.org | also infra-prod-remote-puppet-else failed | 17:48 |
| @clarkb:matrix.org | gitea09 is handling all of the traffic now as the first to update | 17:48 |
| @clarkb:matrix.org | which is good because the service is back up and things seem to generally work. Once the deployment is done I'll confirm with haproxy stats that it thinks every backend is up and then I'll check the log path from frontend to backend and double check that the proxy protocol config is giving us the logging we were hoping for | 17:49 |
| @clarkb:matrix.org | one more backend to update then we're done | 17:55 |
| @clarkb:matrix.org | gitea job reports success and I see 6 up backends for https and 6 up backends for http | 17:57 |
| @clarkb:matrix.org | I have traced a connection from haproxy -> apache -> anubis. Its actually difficult to find random logs that go all the way to gitea due to the volume of bot traffic. May just have ot look at my own IP address | 18:00 |
| @clarkb:matrix.org | that works. I was able to trace a connection I initiated from haproxy -> apache -> anubis -> gitea access log using my source IP | 18:03 |
| @clarkb:matrix.org | so I think this is working and debugging connectivity issues and potentially reenabling the anubis honeypot should be much happier now | 18:04 |
| @clarkb:matrix.org | Anil Belur: fyi ^ your cleanup of my PROXY protocol change is deployed and seems to be working. Thank you for digging into the issues there as this is a nice quality of life improvement | 18:04 |
| @clarkb:matrix.org | and the feared short outage was a reality so I'm glad we waited for a more quiet time period to deploy this | 18:06 |
| @fungicide:matrix.org | i was semi-following along but not typing | 19:06 |
| @fungicide:matrix.org | seems like it went well | 19:06 |
| @fungicide:matrix.org | okay, so haproxy encodes your client address via proxy-protocol to apache, then apache injects it as x-forwarded-for which conveys through anubis to gitea? | 19:07 |
| @fungicide:matrix.org | you didn't need to map up source/destination ports and timestamps to find your request at the backend? | 19:08 |
| @clarkb:matrix.org | correct | 19:14 |
| @clarkb:matrix.org | you may still need to use timestamps a bit if there are multiple requests from the asme client over time. | 19:15 |
| @clarkb:matrix.org | but in general you can grep the ip address and get things mapped front to back and vice versa | 19:15 |
| @fungicide:matrix.org | too awesome. thanks to everyone involved in getting that working! | 19:46 |
| @fungicide:matrix.org | and hopefully it gives anubis a better accounting of abusive clients now that they're not all appearing to originate from the load balancer | 19:46 |
| @fungicide:matrix.org | looking into why `infra-prod-remote-puppet-else` failed, it was trying to access https://opendev.org/opendev/puppet-bup/ at the time that all the gitea backends were probably marked down in haproxy, so i don't think there's anything we need to fix there | 19:52 |
| @fungicide:matrix.org | the task started at 17:47:04 utc and ran for one second | 19:53 |
| @clarkb:matrix.org | Makes sense that the opendev outage would have side effects like that | 20:59 |
| @clarkb:matrix.org | I got immediate errors in my browser too. Not a slow timeout | 20:59 |
| @fungicide:matrix.org | yeah, expected behavior in retrospect, i just wanted to make sure it wasn't indicative of some unrelated problem | 21:02 |
Generated by irclog2html.py 4.1.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!