| -@gerrit:opendev.org- Zuul merged on behalf of Michal Nasiadka: [opendev/system-config] 999383: Add infra-prod-prometheus https://review.opendev.org/c/opendev/system-config/+/999383 | 06:19 | |
| -@gerrit:opendev.org- Michal Nasiadka proposed: [opendev/system-config] 999795: Add infra-prod-service-node-exporter job https://review.opendev.org/c/opendev/system-config/+/999795 | 08:46 | |
| -@gerrit:opendev.org- Michal Nasiadka proposed: [opendev/system-config] 999796: Fix goaccess jobs - update static ssh host key https://review.opendev.org/c/opendev/system-config/+/999796 | 08:57 | |
| @mnasiadka:matrix.org | Clark/fungi: I think I need guidance how to use edit-secrets script (or at least the password to gpg) | 09:24 |
|---|---|---|
| @mnasiadka:matrix.org | Ok, prometheus and greptimedb are running on prometheus01 | 12:51 |
| @mnasiadka:matrix.org | Once 999795 merges we should have a node exporter there to check if metrics end up in VexxHost S3 | 12:56 |
| @fungicide:matrix.org | i'm disappearing to run lunch errands, when i return i'll approve and monitor the anubis upgrade, which should wrap up before we get into the gerrit upgrade timeframe | 15:03 |
| @clarkb:matrix.org | sounds good. I'm catching up on stuff now | 15:09 |
| @clarkb:matrix.org | I've gone ahead and approved https://review.opendev.org/c/opendev/system-config/+/998741 as I expect fungi should be back soon ish and that will take about an hour to gate | 16:30 |
| @clarkb:matrix.org | that is the anubis upgrade change | 16:31 |
| @fungicide:matrix.org | yeah, back, i was just digging up the change number but that seems to be th eone | 16:31 |
| @fungicide:matrix.org | thanks! | 16:31 |
| @harbott.osism.tech:regio.chat | can someone check my comment on https://review.opendev.org/c/openstack/project-config/+/961499/comment/476ae21b_acbf9f3f/ regarding the legacy role? do we need to replace it (and does someone know how), or are we fine with keeping it in use (for now?) | 16:33 |
| @clarkb:matrix.org | I think that the idea is modern zuul will checkout upper constraints properly if you add the repo as a required project to the job? | 16:40 |
| @clarkb:matrix.org | It's possible we can drop that entire role | 16:40 |
| @fungicide:matrix.org | for context, that seems to be specifically about the translation jobs | 16:40 |
| @fungicide:matrix.org | arnaudm: in case you missed my comments earlier, on monday ovh sent us a notice that we have an invoice pending payment for `PublicCloudProject: 8bbdc6f25ee94568baea4e872f032f14 (openstackci)` with invoice id `FR79702243` (i haven't received warnings about our other project yet, and can't recall whether both vouchers typically expire at the same time) | 17:22 |
| @clarkb:matrix.org | fungi: should we send an email to arnaudm maybe? | 17:28 |
| @clarkb:matrix.org | dmsimard may be able to help too | 17:29 |
| @fungicide:matrix.org | oh right!!! how did i forget dmsimard is there now? | 17:30 |
| @dmsimard:matrix.org | hi o/ | 17:35 |
| @dmsimard:matrix.org | Arnaud has been on PTO, I'll figure something out, thanks for the ping | 17:36 |
| @fungicide:matrix.org | thanks as always dmsimard and sorry i didn't think to reach out to you first! | 17:41 |
| @fungicide:matrix.org | this is the time of year for pto, especially in fr | 17:42 |
| @dmsimard:matrix.org | oh yeah for sure, lots of people gone for 3 weeks+ at the same time | 17:51 |
| @fungicide:matrix.org | openmetal has sent us an e-mail notification that we need to enable 2fa for our account on their web portal by the end of next month, if any of our sysadmins finds a few minutes for that | 18:11 |
| @fungicide:matrix.org | er, end of month after next. by november 1 | 18:11 |
| @clarkb:matrix.org | That anubis change is taking forever | 18:14 |
| @fungicide:matrix.org | yeah, i didn't know how long to expect which is why i was hesitant to approve it right before leaving to run errands | 18:20 |
| @fungicide:matrix.org | but i probably could have and it still wouldn't have merged by the time i got home again | 18:21 |
| -@gerrit:opendev.org- Zuul merged on behalf of Clark Boylan: [opendev/system-config] 998741: Update anubis to 1.26.2 https://review.opendev.org/c/opendev/system-config/+/998741 | 18:30 | |
| @fungicide:matrix.org | yay! | 18:32 |
| @clarkb:matrix.org | both lists and gitea deployments are running now | 18:32 |
| @fungicide:matrix.org | yep, well clear of hourly jobs this time | 18:33 |
| @fungicide:matrix.org | i don't anticipate any problems, 1.26.2 has been runing on the wiki server for about a week already | 18:33 |
| @fungicide:matrix.org | anubis has made a surprising difference for mediawiki crawler load | 18:34 |
| @fungicide:matrix.org | the server used to just be entirely unresponsive for hours a day | 18:34 |
| @fungicide:matrix.org | now it's consistently snappy for mew | 18:34 |
| @fungicide:matrix.org | for me | 18:34 |
| @clarkb:matrix.org | ya anubis seems to do just enough to slow the crawlers down | 18:35 |
| @fungicide:matrix.org | i think i'm going to give 1.27.0.pre3 a try on one of my personal web servers | 18:35 |
| @clarkb:matrix.org | lists reports the expected version now. But still waiting for mailman to start up again | 18:36 |
| @clarkb:matrix.org | I think mailman depends on anubis which is why it got restarted too | 18:36 |
| @clarkb:matrix.org | maybe we want to decouple that more? I dunno | 18:37 |
| @clarkb:matrix.org | gitea now too (but gitea deployment isn't complete yet_ | 18:38 |
| @fungicide:matrix.org | there was a thread today on the mailman-users list about using anubis and one of the mm maintainers was talking about how a downside is that it makes the sites inaccessible for browsers without js, but actually i can connect to our sites with lynx no problem it's actually hyperkitty's use of js that makes it impossible to browse list archives | 18:38 |
| @fungicide:matrix.org | anubis routes clients to the js challenge based on user agent strings (among other determining factors) | 18:39 |
| @fungicide:matrix.org | so it lets most typical non-js browsers through unimpeded | 18:39 |
| @clarkb:matrix.org | yes it is configurable. You can in theory have user agents that don't get filtered like that. Until the bots use those agents and need to be filtered anyway | 18:39 |
| @clarkb:matrix.org | the anubis code base upstream has a bunch of configuration examples too fwiw | 18:40 |
| @clarkb:matrix.org | our config is quite simple, but you can get pretty crazy with it. Some of the examples do similar to what we've done elsewhere (IP block based denials, old client UA denials, etc) | 18:40 |
| @clarkb:matrix.org | fungi: once this is done I'm going to find lunch. If you still think today is good for gerrit we can probably approve that change next? | 18:41 |
| @clarkb:matrix.org | looks like it needs to be reviewed too | 18:42 |
| @clarkb:matrix.org | https://review.opendev.org/c/opendev/system-config/+/994938 | 18:42 |
| @clarkb:matrix.org | the big change other than the version bump is switching the replication plugin from a branch to a tag now that the fix we need is on tagged releases | 18:42 |
| @clarkb:matrix.org | gitea deployment has succeeded now too | 18:43 |
| @fungicide:matrix.org | ah yeah should be fine, i'll take a quick look over 994938 and then send out the one-hour status notice | 18:47 |
| @fungicide:matrix.org | #status notice The Gerrit service on review.opendev.org will be offline momentarily at 20:00 UTC (an hour from now) while we restart for a patch upgrade, but should return within a few minutes. | 19:01 |
| @status:opendev.org | @fungicide:matrix.org: sending notice | 19:01 |
| -@status:opendev.org- NOTICE: The Gerrit service on review.opendev.org will be offline momentarily at 20:00 UTC (an hour from now) while we restart for a patch upgrade, but should return within a few minutes. | 19:04 | |
| @status:opendev.org | @fungicide:matrix.org: finished sending notice | 19:04 |
| @clarkb:matrix.org | fungi: looks like gerrit image builds are failing now | 19:10 |
| @fungicide:matrix.org | argh | 19:10 |
| @clarkb:matrix.org | `ERROR: error loading package 'plugins/zuul-results-summary/web': Unable to find package for @@aspect_rules_ts//ts:defs.bzl: The repository '@@aspect_rules_ts' could not be resolved: Repository '@@aspect_rules_ts' is not defined.` | 19:10 |
| @fungicide:matrix.org | tf | 19:10 |
| @clarkb:matrix.org | I don't think I'm fixing that quickly enough to make our targetted time. So I'm just goign to eat lunch and look into it later | 19:10 |
| @clarkb:matrix.org | my guess is they removed some dep from the main repo so now it isn't available to plugins | 19:10 |
| @fungicide:matrix.org | yeah, let's regroup and plan for another attempt later | 19:11 |
| @clarkb:matrix.org | seems like there was a similar issue I ran into at some point | 19:11 |
| @clarkb:matrix.org | but I'll dig in after lunch | 19:11 |
| @fungicide:matrix.org | #status notice The Gerrit service on review.opendev.org will not be restarted at 20:00 UTC after all, due to unanticipated image build regressions, and will be rescheduled to a later time/date. | 19:12 |
| @status:opendev.org | @fungicide:matrix.org: sending notice | 19:12 |
| -@status:opendev.org- NOTICE: The Gerrit service on review.opendev.org will not be restarted at 20:00 UTC after all, due to unanticipated image build regressions, and will be rescheduled to a later time/date. | 19:15 | |
| @status:opendev.org | @fungicide:matrix.org: finished sending notice | 19:15 |
| @fungicide:matrix.org | yeah, the 3.14 build is also failing the same way | 19:16 |
| @dmsimard:matrix.org | Hey, just following up. I don't know much about this billing stuff 😅 I did add a new voucher but I see that the invoice is still pending. I reached out to billing for more info but will likely only hear back tomorrow. | 19:27 |
| @clarkb:matrix.org | Thank you for looking into it | 19:36 |
| @fungicide:matrix.org | thanks! | 19:36 |
| @fungicide:matrix.org | and yeah, for some reason once those time-limited vouchers expire the billing department has to reverse the charges, or at least that's what they've done in the past | 19:37 |
| @fungicide:matrix.org | they also have a fairly short fuse that disables our resources within a few days if they don't reverse the "overdue" charges fast enough, so we should be on the lookout in particular for job log upload failures to swift since i think that's where we'll see it first (manifesting as lots of post_failure job results) | 19:39 |
| @clarkb:matrix.org | https://gerrit.googlesource.com/plugins/zuul-results-summary/+/5f5466265d54345b659459fcdb8957f92215130a this is what broke us | 19:53 |
| @clarkb:matrix.org | I suspect that this works when building against Gerrit master but not 3.13 or 3.14 | 19:53 |
| @clarkb:matrix.org | So now I have to have an awkward conversation upstream asking them why they are breaking the plugin for it's one user | 19:53 |
| @fungicide:matrix.org | yeah... :/ | 19:53 |
| @clarkb:matrix.org | yup confirmed master has a bunch of bazel rules for aspect_rules_ts and stable-3.14 and stable-3.13 have none | 19:58 |
| @clarkb:matrix.org | so upstream updated the plugin to make it work with master and broke us in the process | 19:58 |
| -@gerrit:opendev.org- Clark Boylan proposed: [opendev/system-config] 999904: Checkout working zuul-results-summary commit for stable Gerrit https://review.opendev.org/c/opendev/system-config/+/999904 | 20:15 | |
| @clarkb:matrix.org | This is similar to the hack I had set up with its-base when that broke in a similay way (updates that were master compatible only). I've asked on discord what the proper solution is (I'm guessing stable branches for the plugin) | 20:15 |
| @jim:acmegating.com | stable branches upstream and/or a temporary pin downstream sounds good to me. | 20:20 |
| @jim:acmegating.com | not great that it broke the old versions, but also, maybe kind of nice of them to try to keep it updated for new versions? :) | 20:21 |
| @clarkb:matrix.org | ya and maybe we should do third party ci or something | 20:23 |
| @clarkb:matrix.org | we could run our image builds and system-config-run jobs against changes to upstream things we care about maybe | 20:23 |
| @jim:acmegating.com | we could. we would need to use the deprecated checks plugin; i don't see why that wouldn't work, theoretically, as long as they continue to run it (we would be in the same boat as gerrit's zuul). | 20:25 |
| @clarkb:matrix.org | oh right i forgot about them doing things differently | 20:27 |
| @mnaser:matrix.org | Has there been any changes inside opendev.org filtering things in some way? | 23:42 |
| I'm seeing our CI jobs fail with this: | ||
| fatal: https://opendev.org/openstack/hacking/info/refs not valid: could not determine hash algorithm; is this a git repository? | ||
| @mnaser:matrix.org | (downstream CI jobs though.. on pre-commit run) | 23:42 |
| @mnaser:matrix.org | https://paste.opendev.org/show/bLwbPyj6wCdVzl9bsJ5k/ | 23:42 |
| @clarkb:matrix.org | we upgraded gitea the other day | 23:43 |
| @clarkb:matrix.org | and upgraded anubis this morning | 23:43 |
| @clarkb:matrix.org | but no intentional config updatse as far as I know | 23:43 |
| @mnaser:matrix.org | it seems non-deterministic .. so i wonder if its the scraping filters or something that caught this | 23:43 |
| @clarkb:matrix.org | it seems to work fine in my browser (which has cleared anubis) and with curl which I think is what git will use (but with different user agent) | 23:46 |
| @clarkb:matrix.org | does the log file it indicates you should check include any more info (like maybe an http return code or byte size or something) | 23:47 |
| @clarkb:matrix.org | but also this is why I hate precommit | 23:47 |
| @clarkb:matrix.org | apache logs on each gitea backend report requests to hacking info are all 200 responses | 23:51 |
| @mnaser:matrix.org | let me try and jump into ssh during aa run | 23:51 |
| @clarkb:matrix.org | this is interseting though. Some of those 200 responses are only 172 bytes long | 23:55 |
| @clarkb:matrix.org | I see others in the 3800 range whcih more closely matches what I get when I request it | 23:55 |
| @mnaser:matrix.org | maybe thats the first intial anubis response | 23:55 |
| @clarkb:matrix.org | well anubis should ignore your client I think because its git | 23:55 |
| @mnaser:matrix.org | ``` | 23:58 |
| runner@depot-worker-x7kpnhxc47zfgbqjwwh2:/home/runner/work/ironic/ironic$ git clone https://opendev.org/openstack/hacking | ||
| Cloning into 'hacking'... | ||
| fatal: https://opendev.org/openstack/hacking/info/refs not valid: could not determine hash algorithm; is this a git repository? | ||
| ``` | ||
| @clarkb:matrix.org | ok the 172 byte response seems to be acceptable. I was able to reproduce it with git locally against gitea10 | 23:58 |
| @mnaser:matrix.org | source ip is 44.212.135.8 | 23:59 |
| @clarkb:matrix.org | there must be some optimization in git on the client side sending the refs it already has and get back a diff or something | 23:59 |
| @clarkb:matrix.org | but when you do it with curl or the browser you get all the refs back | 23:59 |
Generated by irclog2html.py 4.1.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!