| opendevreview | Ian Wienand proposed openstack/openstack-zuul-jobs master: flake8 : ignore .cache subdirectory https://review.opendev.org/c/openstack/openstack-zuul-jobs/+/881938 | 02:06 |
|---|---|---|
| opendevreview | Ian Wienand proposed openstack/openstack-zuul-jobs master: flake8 : ignore .cache subdirectory, avoid broken ansible-compat https://review.opendev.org/c/openstack/openstack-zuul-jobs/+/881938 | 02:50 |
| opendevreview | Merged openstack/openstack-zuul-jobs master: flake8 : ignore .cache subdirectory, avoid broken ansible-compat https://review.opendev.org/c/openstack/openstack-zuul-jobs/+/881938 | 03:03 |
| opendevreview | Merged openstack/openstack-zuul-jobs master: Fix installation of py27 on CentOS 7/8 https://review.opendev.org/c/openstack/openstack-zuul-jobs/+/881890 | 04:50 |
| *** elvira1 is now known as elvira | 07:03 | |
| *** dviroel__ is now known as dviroel | 11:36 | |
| opendevreview | Ashutosh Sarode proposed openstack/project-config master: Add Harbor app to starlingx https://review.opendev.org/c/openstack/project-config/+/881960 | 11:37 |
| opendevreview | Ashutosh Sarode proposed openstack/project-config master: Add Harbor app to StarlingX https://review.opendev.org/c/openstack/project-config/+/881962 | 12:01 |
| opendevreview | Ashutosh Sarode proposed openstack/project-config master: Add Harbor app to StarlingX https://review.opendev.org/c/openstack/project-config/+/881962 | 12:06 |
| opendevreview | Ashutosh Sarode proposed openstack/project-config master: Add Harbor app to StarlingX https://review.opendev.org/c/openstack/project-config/+/881962 | 12:37 |
| opendevreview | Jeremy Stanley proposed openstack/project-config master: linters: avoid broken ansible-compat https://review.opendev.org/c/openstack/project-config/+/881963 | 12:43 |
| *** sfinucan is now known as stephenfin | 13:04 | |
| opendevreview | Ashutosh Sarode proposed openstack/project-config master: Add Harbor app to StarlingX https://review.opendev.org/c/openstack/project-config/+/881962 | 13:14 |
| opendevreview | Merged openstack/project-config master: linters: avoid broken ansible-compat https://review.opendev.org/c/openstack/project-config/+/881963 | 13:26 |
| opendevreview | Merged openstack/project-config master: Retire puppet-tacker - Step 5: Remove Project https://review.opendev.org/c/openstack/project-config/+/875291 | 13:57 |
| opendevreview | Merged openstack/project-config master: Retire puppet-rally - Step 1: End project Gating https://review.opendev.org/c/openstack/project-config/+/879419 | 14:00 |
| noonedeadpunk | folks, are you aware of any issues with gitea backends? | 14:12 |
| noonedeadpunk | As we see in CI periodical issues with interacting to it, ie `Request failed: <urlopen error _ssl.c:1114: The handshake operation timed out>", "url": "https://releases.openstack.org/constraints/upper/2949b08eeb90ff664bd2312d994367f8f8b491ae"` | 14:12 |
| noonedeadpunk | Also I see random failures when accessing opendev.org from browser | 14:13 |
| noonedeadpunk | sometimes it takes like 6-20 sec to load the page | 14:15 |
| fungi | noonedeadpunk: that's not anything to do with gitea | 14:20 |
| noonedeadpunk | aha, ok | 14:20 |
| fungi | the gitea urls all start with https://opendev.org/ | 14:20 |
| noonedeadpunk | well, it redirects to gitea | 14:20 |
| fungi | releases.opendev.org is a staticly published website served from data in afs (like docs.openstack.org et al) | 14:20 |
| fungi | any idea if the handshake is failing while fetching from releases.o.o or after redirection? | 14:21 |
| noonedeadpunk | and I also saw in browser some tls-related issues periodically when using opendev.org | 14:21 |
| noonedeadpunk | unfortunatelly, not :( | 14:21 |
| noonedeadpunk | we catch that in CI like https://zuul.opendev.org/t/openstack/build/2ea75356a2294738b260ba85f2a424e5/log/job-output.txt#10740 | 14:22 |
| noonedeadpunk | so no more data available there. And ofc it's highly intermittent | 14:22 |
| fungi | the releases.o.o site is served from rackspace's dfw region, while opendev.org is in vexxhost's sjc1 region | 14:23 |
| noonedeadpunk | Like 1 out of 20 job is failing | 14:23 |
| noonedeadpunk | iirc I should check for headers to see the gitea backend? | 14:24 |
| fungi | also note that the releases.openstack.org constraints redirector isn't intended for use by ci jobs, it's there to ease local testing by devs since the tox configs can refer them to retrieve the constraints files consistently. ci jobs would ideally set openstack/requirements as a required-project and then use the supplied copy of constraints on the test node | 14:24 |
| fungi | what's making the urlopen call, and could it be more verbose about response codes and redirects it's following? | 14:26 |
| noonedeadpunk | I think we have a bug due to which we use not zuul-provided repo despite all set to use it... | 14:27 |
| fungi | ideally we'd work out whether it's getting the handshake timeout from releases.o.o pre-redirection vs from opendev.org haproxy/gitea post-redirect | 14:27 |
| noonedeadpunk | We were thinking that we're using local copy but seems we're not somehow... | 14:28 |
| noonedeadpunk | I've already pushed huge refactoring patch that covers bug https://review.opendev.org/c/openstack/openstack-ansible/+/881824 | 14:29 |
| noonedeadpunk | so yeah, I know we should not used that at the first place :( | 14:30 |
| noonedeadpunk | I will try to reproduce that in browser now to track down the backend | 14:30 |
| noonedeadpunk | As I bet I saw issues today/tomorrow with just opendev.org | 14:30 |
| fungi | the response headers probably mention the backend name, but the cn on the ssl cert will too | 14:39 |
| opendevreview | Ashutosh Sarode proposed openstack/project-config master: Add Harbor app to StarlingX https://review.opendev.org/c/openstack/project-config/+/881962 | 15:02 |
| *** Guest74 is now known as atmark | 15:22 | |
| noonedeadpunk | so I just get 20s response time while jsut connecting to gitea14.opendev.org | 15:23 |
| noonedeadpunk | but no timeouts yet.... | 15:24 |
| opendevreview | Jeremy Stanley proposed openstack/project-config master: Fix the "all" transformation so it actually works https://review.opendev.org/c/openstack/project-config/+/882075 | 15:25 |
| clarkb | 09-13 seem to be quick but I agree 14 seems to be slow to respond. We can take 14 out of the rotation (at the risk that the load balancer will shift whatever is creating this problem to anothe backend) or restart it under the assumption it is something a restart would correct. Probably need to look more closely at logs before decided (maybe we have a bad we crawler at work | 15:27 |
| clarkb | again) | 15:27 |
| fungi | noonedeadpunk: interesting, its resource graphs don't look too wild at least: http://cacti.openstack.org/cacti/graph_view.php?action=tree&tree_id=1&leaf_id=1161&nodeid=node1_1161&host_group_data= | 15:27 |
| clarkb | fungi: ya I agree all of the giteas look fine in cacti but the slowness is observable. Good chance its just doing its best to respond to a bad crawler we need to block | 15:27 |
| noonedeadpunk | well, it sounds a bit like connection-persistance or smth like that | 15:31 |
| noonedeadpunk | As once connection is established - all following reuests are kinda fast | 15:31 |
| noonedeadpunk | until you idle for 5 mins and try to reload page again | 15:31 |
| clarkb | I realoaded and got slightly faster but not expected speed so not sure I'm observing that | 15:31 |
| opendevreview | Ashutosh Sarode proposed openstack/project-config master: Add Harbor app to StarlingX https://review.opendev.org/c/openstack/project-config/+/881962 | 15:31 |
| clarkb | but ya it could be the apache in front is simply failing to rotate out connections. That could be due to a web crawler | 15:32 |
| clarkb | I need to load keys and can take a look | 15:32 |
| clarkb | yes I think we have a crawler | 15:39 |
| clarkb | This is really getting old. Looks like a number of newer user agents too. I'll work on a change | 15:40 |
| clarkb | they can see all this info much more quickly if they just git clone | 15:41 |
| clarkb | but these bots don't seem to be designed to be friendly | 15:41 |
| clarkb | fungi: re ^ there are some UAs that are clearly filterable. Others appear to potentially belong to modern clients on older (but still supported?) iphones as well as apps like wechat on iphone | 15:52 |
| clarkb | should we block them anyway? or start with a subset and see where that gets us? | 15:53 |
| clarkb | hrm some of these are for older iphones that are not updated even though the major version has newer releases. I think we should block those | 15:53 |
| opendevreview | Jeremy Stanley proposed openstack/project-config master: Add an "apply" transformation which applies all https://review.opendev.org/c/openstack/project-config/+/882075 | 15:54 |
| opendevreview | Jeremy Stanley proposed openstack/project-config master: Make option indenting a selectable transformation https://review.opendev.org/c/openstack/project-config/+/882080 | 15:54 |
| fungi | clarkb: yeah, i'd just add the obvious ones to the ua filter | 15:55 |
| noonedeadpunk | clarkb: regarding UAs - could it be some OSA DOS we've observed previously? | 15:55 |
| noonedeadpunk | Like just pulling tons of repos from the same subnet? | 15:55 |
| fungi | we normally see a different resource usage pattern when that happens | 15:56 |
| noonedeadpunk | As I might know who that could be... | 15:56 |
| fungi | it ends up eating a ton of resident memory to hold all the in flight copies of the nova repo and such | 15:56 |
| fungi | though maybe that has changed with recent gitea updates | 15:56 |
| clarkb | noonedeadpunk: no this isn't OSA | 15:56 |
| fungi | i wouldn't expect it to | 15:56 |
| clarkb | unless OSA is doing silly things with user agents and gitea | 15:57 |
| fungi | osa added a clear user agent a while back, specifically so that we can spot it if we run into that situation again | 15:57 |
| noonedeadpunk | Well, I can't say we're not doing silly things :D | 15:57 |
| fungi | sometimes i've done as many as six silly things before breakfast | 15:58 |
| noonedeadpunk | In case these requests are from 31.131.16.0/20 - I know who that is and can reach them, just in case | 15:58 |
| fungi | it sounds like this is one of the distributed crawler botnets we keep seeing that tries to evade detection by using randomized user agents | 16:01 |
| noonedeadpunk | ok, yes, then it;s unlikely the folks I know | 16:02 |
| clarkb | I've got a change with like 15 or 20 new UAs to filter. Now trying to be better with grep and sort and uniq to capture any I missed | 16:05 |
| clarkb | probably should've started there but I didn't realize it would be so many (in the past it hasn't been) | 16:05 |
| clarkb | ok I'm glad I did that as now I have a pretty definitive list | 16:15 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!