| Mahdey | Hi, I have a quick question: my patch adds `entered_at` and `exited_at` to the node history API response, should I add a new microversion for them? | 00:16 |
|---|---|---|
| Mahdey | Also, right now I record deployment start and deployment end as two separate history entries. Is that okay, or should it be one entry with both timestamps? | 00:16 |
| pas-ha[m] | gitea on opendev.org is down with SSL errors (again?) curl: (35) TLS connect error: error:0A000126:SSL routines::unexpected eof while reading | 11:36 |
| pas-ha[m] | maybe false alarm ^, looks more like some network troubles with some of providers that my requests routes via | 11:47 |
| *** profcorey1 is now known as profcorey | 15:48 | |
| -opendevstatus- NOTICE: Load on the opendev.org Gitea backends is under control again for now, if any Zuul jobs failed with SSL errors or disconnects reaching the service prior to 16:15 UTC they can be safely rechecked | 17:03 | |
| Mike-- | opendev.org throws 403 here for all urls, known? | 17:42 |
| *** dmellado471 is now known as dmellado47 | 17:48 | |
| frickler | Mike--: we needed to block some AI crawlers, it is possible that we overdid it, can you try with a different browser? or were your errors from earlier today? | 18:13 |
| Mike-- | Google Chrome on Mac | 18:19 |
| Mike-- | will try another | 18:19 |
| Mike-- | same with safari | 18:20 |
| Mike-- | works from another laptop | 18:20 |
| Mike-- | so I would guess it's a bit too strict (other laptop = windows) | 18:20 |
| Mike-- | that's same source IP, just different platform (macos vs. windows) | 18:22 |
| frickler | fungi: ^^ can you check this please? I'm eod-ing now | 18:22 |
| Mike-- | both google chrome (if that helps) :) | 18:23 |
| fungi | Mike--: any chance it's an outdated chrome version? | 18:24 |
| Mike-- | It says up2date | 18:25 |
| Mike-- | 146.0.7680 | 18:25 |
| Mike-- | 146.0.7680.178 | 18:26 |
| Mike-- | it's an arm based mac (if that makes a difference) | 18:26 |
| clarkb | sorry I don't have scrollback and I think fungi is busy so I'll take a look | 18:27 |
| clarkb | Mike--: can you share a specific url that you've tried to access? It will help me cross reference against our logs | 18:27 |
| clarkb | alternatively can you open https://opendev.org/opendev/system-config/commit/6ac3f503728cab6733e078abf4b5fccf0e990305 | 18:28 |
| Mike-- | https://opendev.org | 18:29 |
| clarkb | Mike--: ok, it is helpful if there is something more specific like the url above as that makes filtering logs easier | 18:29 |
| clarkb | can you open that link (I know it will fail but it should generate logs I can look at) | 18:29 |
| Mike-- | Can't copy/paste (remote desktop limitations) so need to type it over hold on | 18:30 |
| Mike-- | clarkb: done | 18:30 |
| clarkb | cool I'm looking for logs now. Give me a few minutes | 18:31 |
| Mike-- | NP | 18:31 |
| clarkb | ok I see it (for other sysadmins the balancer seems to send Mike-- to gitea14) | 18:32 |
| clarkb | Mike--: according to our logs you're on an EOL OS X version on Intel not arm | 18:33 |
| Mike-- | LOL? | 18:33 |
| Mike-- | Apple M4 Pro Tahoe 26.4 | 18:33 |
| clarkb | but also that User Agent string matches one we've had to block due to its signifant usage by crawlers creating problems. We were operating under the assumption that anything on the EOL OS X version could be dropped as invalid | 18:35 |
| Mike-- | I would say yes to your assumption, but it's a company laptop that has extreme update control. It's up2date as it can be | 18:36 |
| clarkb | I'm trying to pull up more info for you. But the next step may be to have you open chrome's developer tools then look in the network tab to find the user agent you are sending with your requests | 18:36 |
| Mike-- | as such I'd say your OS EOL check needs a tweak | 18:36 |
| clarkb | Mike--: well maybe I mean we're doing what we can here. The alternative is that the service could be down for everyone else too | 18:36 |
| Mike-- | I'm not complaining, trying to think along | 18:36 |
| clarkb | I'm just trying to pull up where we added the rules to dobule check that this is why it is happening | 18:37 |
| clarkb | but also if you go to chrome's menu -> more tools -> developer tools it should open a panel with a network tab. iF you open the network tab then refresh on https://opendev.org you can see the requests you send. Click on a request and going to headers you should see the User-Agent sent | 18:37 |
| Mike-- | user agent I see = "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/146.0.0.0 Safari/537.36 | 18:38 |
| clarkb | cool that matches. Any idea why it indicates it is OS 10_15_7 and intel when you say you are on arm and running tahoe? | 18:39 |
| clarkb | note 10_15_8 is the latest (and last version of catalina aiui) | 18:39 |
| clarkb | a potentially simple solution here is for your browser to be accurate | 18:39 |
| Mike-- | if I read online the user-agent is correct and latest | 18:40 |
| Mike-- | https://www.whatismybrowser.com/guides/the-latest-user-agent/macos | 18:40 |
| clarkb | thats weird why would it say it is intel? | 18:40 |
| Mike-- | I suspect the Macintosh before that is the main target | 18:41 |
| Mike-- | I honestly don't know | 18:41 |
| clarkb | anyway the rule we added was for "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/137.0.0.0 Safari/537.36" which isn't a match | 18:41 |
| Mike-- | but my coworker has the same | 18:41 |
| clarkb | so now I'm digging more to see why that particular string is matching (it could be another rule unexpected) | 18:41 |
| clarkb | I'm guessing but don't know for sure yet that maybe this was one of the ad hoc rules fungi added today | 18:42 |
| clarkb | so I'm following up on that now | 18:42 |
| fungi | entirely likely | 18:43 |
| clarkb | yes that appears to be the case | 18:43 |
| clarkb | I'm going to remove that single entry and reload apaches which should fix this | 18:43 |
| clarkb | and then TIL arm macs say they are intel macs because who knows why | 18:43 |
| fungi | i was dredging the request logs for the user agents of anything hitting /commit/ paths on the gitea server since we were being flooded with them, so am not surprised i might have caught up some legitimate browsers' requests in the mix | 18:44 |
| clarkb | I'm also going to drop the rule for chome 147 | 18:44 |
| fungi | or the bots were spoofing legitimate current user agents | 18:44 |
| fungi | clarkb: fine by me, thanks! | 18:44 |
| Mike-- | https://nielsleenheer.com/articles/2024/trouble-happens-when-you-update-a-version-number/ | 18:44 |
| Mike-- | I facepalmed when I read this, I suspect you will too | 18:44 |
| fungi | hah | 18:44 |
| fungi | i palm is already permanently stapled to my forehead anyway | 18:45 |
| fungi | er, my palm | 18:45 |
| Mike-- | basically it states the the first part is uselss and you need to look at the Chrome/(version) part only.... :( | 18:45 |
| clarkb | Mike--: I've updated gitea14 which I think is the backend you'll get load balacned to. Can you test again? if it still doesn't work then we'll wait for me to do the other backends and try again then | 18:46 |
| Mike-- | and to answer your other remark: with python selenium it is dead easy to mimic a recent browser | 18:46 |
| Mike-- | clarkb: it works yes | 18:46 |
| clarkb | Mike--: right the problem is we're under consistent active ddos by massive botnets trolling the internet for data to presumably feed into llm training sets. They are spoofing many many many obviously wrong user agents | 18:46 |
| clarkb | we've managed to keep things somewhat under control by actively removing requests for things that were obviously wrong. Things like modern crhome on windows 98 | 18:47 |
| clarkb | or ancient chrome on latest windows | 18:47 |
| Mike-- | you might be aware but imdb.com (AWS) has similar issues and engaged in extreme WAF breaking imdb.com occassionally over the last months | 18:47 |
| Mike-- | clarkb: seriously recent chrome on windows 98? LOL | 18:47 |
| clarkb | unfortunately today things got bad again and there was some quick updates to try and stop the bleeding and your specific user agent got caught up in that. I'm cleaning that up now | 18:47 |
| clarkb | Mike--: we've seen agents as far back as windows 95 yes | 18:48 |
| clarkb | and android 1 | 18:48 |
| clarkb | and so on | 18:48 |
| clarkb | anyway I'm going to edit the other backends no | 18:48 |
| clarkb | *now | 18:48 |
| Mike-- | thanks so much, I also use the github synced repo's | 18:48 |
| Mike-- | but still thought it worth to mention hee | 18:48 |
| Mike-- | *here | 18:48 |
| Mike-- | might not be the only one | 18:48 |
| Mike-- | s/might/I might/ | 18:49 |
| clarkb | yup thank you for reporting it. This was definitely in error and its good for us to fix it | 18:50 |
| clarkb | Mike--: oh also we've found some fun typos in the strings like humans are curating them and making mistakes | 18:51 |
| Mike-- | I can imagine | 18:51 |
| clarkb | not really relevant here, its just interesting how this ends up happening in practice | 18:51 |
| fungi | absolutely, thanks for bringing it up and my apologies for the accidental false-positive | 18:51 |
| Mike-- | No worries, nothing broke and I could use other sources (github, etc) | 18:51 |
| fungi | i was working in haste, scrambling to get the systems back on line again | 18:52 |
| Mike-- | community++ ! | 18:52 |
| clarkb | ok all the backends should have 146 and 147 removed as those seem valid. and good to know that modern osx even on arm reports it is old osx on intel | 18:52 |
| Mike-- | thanks again | 18:53 |
| Mike-- | appreciate the effort everyone puts in! | 18:53 |
Generated by irclog2html.py 4.1.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!