| @mnasiadka:matrix.org | corvus: there’s a task in post that does df - you can see full /opt there | 05:07 |
|---|---|---|
| -@gerrit:opendev.org- Michal Nasiadka proposed: [opendev/zone-opendev.org] 983860: Add entries for mirror03.ord.rax https://review.opendev.org/c/opendev/zone-opendev.org/+/983860 | 12:53 | |
| -@gerrit:opendev.org- Zuul merged on behalf of Michal Nasiadka: [opendev/zone-opendev.org] 983851: Promote mirror04.gra1.ovh https://review.opendev.org/c/opendev/zone-opendev.org/+/983851 | 13:09 | |
| @fungicide:matrix.org | mnasiadka: yes, also useful but doesn't get us disk usage trends over the course of the job | 13:22 |
| @fungicide:matrix.org | only the final state | 13:22 |
| @fungicide:matrix.org | 5-minute load average on our gitea backends right now ranges from 0.11 (gitea11) to 1.32 (gitea10) | 13:24 |
| @fungicide:matrix.org | site is up and responsive | 13:25 |
| @fungicide:matrix.org | checking gitea access logs on the backends, a typical mix of requests are making it through anubis just fine | 13:26 |
| -@gerrit:opendev.org- Michal Nasiadka proposed: [opendev/zone-opendev.org] 983860: Add entries for mirror03.ord.rax https://review.opendev.org/c/opendev/zone-opendev.org/+/983860 | 13:30 | |
| @mnasiadka:matrix.org | fungi: that sounds like a promise of at least not that often fire drills | 13:31 |
| @mnasiadka:matrix.org | fungi: DIB creates a raw, qcow2 and vhd in the same run, especially that vhd creation requires quite some amount of storage due to intermediate (and .bak) | 13:32 |
| @mnasiadka:matrix.org | So it might be we were close before, and now with Resolute things got bigger | 13:32 |
| @fungicide:matrix.org | mnasiadka: that's been our suspicion, yes | 13:36 |
| @fungicide:matrix.org | but it's hard to tell just looking at the end state since files may have gotten deleted before then | 13:37 |
| @fungicide:matrix.org | especially if it was something like a huge tempfile that got immediately unlinked when the parent process hit enospc | 13:38 |
| @mnasiadka:matrix.org | I think frickler has held a node for debugging, I was planning to just run the build locally on a VM and check sizings of all image types | 13:43 |
| @fungicide:matrix.org | good plan, curious to hear how it goes | 13:44 |
| @fungicide:matrix.org | just remember that the disk usage at the end may be less than at some points during the build | 13:44 |
| @fungicide:matrix.org | (as in it's teoretically possible, but i don't know if it ever will be in practice) | 13:45 |
| @fungicide:matrix.org | theoretically | 13:45 |
| -@gerrit:opendev.org- Michal Nasiadka proposed: [opendev/zone-opendev.org] 984030: Replace whitespaces with tabs https://review.opendev.org/c/opendev/zone-opendev.org/+/984030 | 13:50 | |
| -@gerrit:opendev.org- Jeremy Stanley https://matrix.to/#/@fungicide:matrix.org marked as active: [openstack/project-config] 977380: Replace 2026.1/Gazpacho key with 2026.2/Hibiscus https://review.opendev.org/c/openstack/project-config/+/977380 | 14:09 | |
| -@gerrit:opendev.org- Zuul merged on behalf of Jeremy Stanley https://matrix.to/#/@fungicide:matrix.org: [openstack/project-config] 977380: Replace 2026.1/Gazpacho key with 2026.2/Hibiscus https://review.opendev.org/c/openstack/project-config/+/977380 | 14:20 | |
| @clarkb:matrix.org | I have to do a school run. Have we landed an acl change yet to exercise manage-projects yet? | 14:22 |
| @clarkb:matrix.org | I think that is a good idea if we have one. Then if that looks good we can go for full project creation | 14:23 |
| @fungicide:matrix.org | not yet, no, i was checking on the health of things and wrapping up a meeting | 14:23 |
| @clarkb:matrix.org | I guess the daily run would've exercises it a bit too | 14:29 |
| @clarkb:matrix.org | If we want to find those logs on bridge | 14:29 |
| @clarkb:matrix.org | https://zuul.opendev.org/t/openstack/build/f49fa46920864ed89f29dd83f891f1d7 this is the daily manage-projects run from today. It succeeded. So ya I think next step is acl update then mordred's new project | 14:46 |
| @mordred:waterwanders.com | huzzah! | 14:46 |
| @clarkb:matrix.org | I have approved https://review.opendev.org/c/openstack/project-config/+/981924 which is the acl update fungi called out yesterday | 14:47 |
| @mnasiadka:matrix.org | Clark: if you'll have some time today - I updated https://review.opendev.org/c/opendev/zone-opendev.org/+/983860 (and posted a patch on top to convert all whitespaces to tabs) | 14:51 |
| @mnasiadka:matrix.org | There are also https://review.opendev.org/c/opendev/system-config/+/983610 and https://review.opendev.org/c/opendev/system-config/+/983911 that are waiting for fungi (or somebody that is not Clark) ;-) | 14:53 |
| @clarkb:matrix.org | mnasiadka: I approved the first DNS update. I might hold off on the tab cleanup for now. Mostly because I'm hoping for an easy straightforward day and big dns refactor (even if it should noop) is maybe not in line with that | 14:55 |
| @mnasiadka:matrix.org | Clark: I just posted that because it was quite easy to do, it can wait for a better day :-) | 14:55 |
| @clarkb:matrix.org | mnasiadka: according to grafana mirror.ubuntu has continued to release after I was done with it. Did this fix the problem you saw with stale packages? | 14:56 |
| @clarkb:matrix.org | Seeing job retries due to failure to connect to the ovh gra1 mirror | 15:00 |
| @clarkb:matrix.org | looks like it tries both ipv4 and ipv6 if I'm reading that correctly | 15:01 |
| -@gerrit:opendev.org- Zuul merged on behalf of Dmitriy Rabotyagov: [openstack/project-config] 981924: Introduce OpenStack-Ansible Power Reviewers group https://review.opendev.org/c/openstack/project-config/+/981924 | 15:01 | |
| @clarkb:matrix.org | mnasiadka: it looks like apache isn't running or configured there | 15:03 |
| @clarkb:matrix.org | did we update DNS before the inventory maybe? | 15:03 |
| @clarkb:matrix.org | yes I think that is what happened | 15:05 |
| @clarkb:matrix.org | I'll push a revert for the DNS update now | 15:05 |
| -@gerrit:opendev.org- Clark Boylan proposed: [opendev/zone-opendev.org] 984049: Revert "Promote mirror04.gra1.ovh" https://review.opendev.org/c/opendev/zone-opendev.org/+/984049 | 15:06 | |
| -@gerrit:opendev.org- Clark Boylan proposed: [opendev/zone-opendev.org] 984049: Revert "Promote mirror04.gra1.ovh" https://review.opendev.org/c/opendev/zone-opendev.org/+/984049 | 15:07 | |
| @clarkb:matrix.org | need to remember that git revert doesn't update the serial in the correct direction | 15:07 |
| @clarkb:matrix.org | https://review.opendev.org/c/opendev/system-config/+/983610 is the change | 15:08 |
| @clarkb:matrix.org | * https://review.opendev.org/c/opendev/system-config/+/983610 is the change to add the server to the inventory and configure it as a mirror | 15:08 |
| @clarkb:matrix.org | I think things are mostly working due to zuul retries. But getting that reviewed and merged quickly is probably a good idea. I'll likely self approve once check testing comes back happy | 15:10 |
| @fungicide:matrix.org | i approved the revert and also the missed inventory addition | 15:13 |
| @clarkb:matrix.org | thanks. Need to monitor it and make sure it doesn't fail as our zone fiel checking appears to install the tool it needs in run not pre-run so won't retry | 15:14 |
| @clarkb:matrix.org | the acl update deployment is running now and shouldn't be affected by this. | 15:14 |
| @clarkb:matrix.org | yup it failed. https://zuul.opendev.org/t/openstack/build/c86fd71abc4346d3b342e609fd6b3342/log/job-output.txt#279 | 15:20 |
| @clarkb:matrix.org | do we want to try force merging it? I think gra1 has lots of capacity because jobs are kicking out quickly there which causes more thinsg to schedule there | 15:20 |
| @clarkb:matrix.org | alternatively I could set max-servers to 0 in gra1, land the revert, then restore gra1 after enough dns cache ttl timeout has happened | 15:21 |
| @clarkb:matrix.org | maybe that is the best approach | 15:21 |
| @fungicide:matrix.org | deploy for the acl update did succeed though | 15:22 |
| @clarkb:matrix.org | cool. That doesn't rely on the mirror in gra1 but good that it is running happily after yesterday's gitea work | 15:22 |
| -@gerrit:opendev.org- Clark Boylan proposed: [opendev/zuul-providers] 984053: Temporarily disable ovh gra1 while we fix its mirror https://review.opendev.org/c/opendev/zuul-providers/+/984053 | 15:23 | |
| @clarkb:matrix.org | that runs noop jobs so should succeed. If we approve that we can then rerun jobs to update dns and reenable the region once caches should be clean | 15:24 |
| @mnasiadka:matrix.org | Clark: re Ubuntu mirror it works fine now | 15:24 |
| @fungicide:matrix.org | i've approved 984053, was going to try to pop out early for lunch right now if things are not really on fire | 15:25 |
| @clarkb:matrix.org | fungi: I think its fine. Once the region is disabled we can take our time to fix it | 15:25 |
| @clarkb:matrix.org | I'll recheck the dns revert as soon a the zuul-providers change lands | 15:25 |
| -@gerrit:opendev.org- Zuul merged on behalf of Clark Boylan: [opendev/zuul-providers] 984053: Temporarily disable ovh gra1 while we fix its mirror https://review.opendev.org/c/opendev/zuul-providers/+/984053 | 15:26 | |
| @mnasiadka:matrix.org | Clark: argh, I need to be better with depends-on - I was pretty sure that mirror is in the inventory | 15:27 |
| @clarkb:matrix.org | ya I think its a good reminder to double check relationships when pushing and when reviewing | 15:27 |
| @fungicide:matrix.org | i'll be back in an hour-ish and can get back to trying to fix the apt-puppetlabs mirroring then | 15:27 |
| @clarkb:matrix.org | sounds good. I think the immediate gra thing should be handled. The dns update should be mergeable after the recheck now and then we can put gra1 back into the rotation in an hour or so. | 15:28 |
| @clarkb:matrix.org | fungi: maybe when you get back we approve mordred's project change. I'll go review it now | 15:28 |
| -@gerrit:opendev.org- Clark Boylan proposed: [opendev/zuul-providers] 984056: Revert "Temporarily disable ovh gra1 while we fix its mirror" https://review.opendev.org/c/opendev/zuul-providers/+/984056 | 15:32 | |
| @clarkb:matrix.org | There's the change to put gra1 back itno the rotation once dns and mirror look happy again | 15:32 |
| -@gerrit:opendev.org- Zuul merged on behalf of Clark Boylan: [opendev/zone-opendev.org] 984049: Revert "Promote mirror04.gra1.ovh" https://review.opendev.org/c/opendev/zone-opendev.org/+/984049 | 15:41 | |
| @mnasiadka:matrix.org | Clark: run static failed in 983610 (launchpad down again?) but the mirror job succeeded | 16:00 |
| @clarkb:matrix.org | Yes looks like ppa access problems | 16:01 |
| @clarkb:matrix.org | I'm able to get the gra mirror now. but will probably wait for another 40 minutes or so before approving the revert to add gra1 back to the pool just to be sure | 16:02 |
| @mnasiadka:matrix.org | Thanks and sorry for the trouble ;) | 16:06 |
| @clarkb:matrix.org | no worries. This si why we start with the mirrors. Its a good space to learn in where the stakes are lower | 16:12 |
| @clarkb:matrix.org | I have approved the change to put gra1 back into the zuul-launcher mix | 16:47 |
| -@gerrit:opendev.org- Zuul merged on behalf of Clark Boylan: [opendev/zuul-providers] 984056: Revert "Temporarily disable ovh gra1 while we fix its mirror" https://review.opendev.org/c/opendev/zuul-providers/+/984056 | 16:47 | |
| @clarkb:matrix.org | https://zuul.opendev.org/t/openstack/stream/04e349cae59849c2a59af98b1224510a?logfile=console.log is running in gra1. I'm following its stream log | 16:52 |
| @clarkb:matrix.org | I think the apt cache update is working. But the task hasn't completed yet | 16:54 |
| @fungicide:matrix.org | approved mordred's 983912 now | 16:55 |
| @fungicide:matrix.org | and yes, lp's matrix room has been abuzz with discussion of ppa problems again today | 16:56 |
| @clarkb:matrix.org | the job I linked to failed, but did so in its own unittests. the whole apt cache update, run bindep, install things with apt and then later pip seem to have worked | 16:59 |
| -@gerrit:opendev.org- Zuul merged on behalf of Monty Taylor https://matrix.to/#/@mordred:inaugust.com: [openstack/project-config] 983912: Add gerrit plugin for openclaw https://review.opendev.org/c/openstack/project-config/+/983912 | 17:08 | |
| @clarkb:matrix.org | manage-projects has just started | 17:10 |
| @clarkb:matrix.org | https://opendev.org/inaugust/openclaw-gerrit exists and has the jeepyb created .gitreview file in it | 17:16 |
| @clarkb:matrix.org | the job reported success as well | 17:16 |
| @clarkb:matrix.org | I think that all checks out to me | 17:16 |
| @fungicide:matrix.org | there's a related one too, https://review.opendev.org/983924 | 17:18 |
| @clarkb:matrix.org | I have approved that one as well | 17:19 |
| @clarkb:matrix.org | with that largely wrapping up the immediate concerns around gitea I think my next focus is Gerrit upgrade prep. fungi have you had a chance to look at the etherpad yet? curious if you've got any feedback or have anything that jumps out at you as needing attention before Sunday? | 17:20 |
| @clarkb:matrix.org | Otherwise I think my plan is to be at the comptuer around 1900 so there is plenty of time to send notices and add things to the emergency file before doing the actual upgrade at 20:00 as announced | 17:21 |
| @fungicide:matrix.org | i believe i did look at the upgrade etherpad, but will do so again since we're 2 days out | 17:26 |
| @fungicide:matrix.org | what's the url, just to be sure i'm not looking at the wrong one? | 17:26 |
| @clarkb:matrix.org | https://etherpad.opendev.org/p/gerrit-upgrade-3.12 | 17:27 |
| @clarkb:matrix.org | and honestly if we can't think of anything that needs to be addressed in ^ before Sunday I may just pop out after lunch. Its been a busy week and I'll be working sunday too | 17:28 |
| -@gerrit:opendev.org- Zuul merged on behalf of Monty Taylor https://matrix.to/#/@mordred:inaugust.com: [openstack/project-config] 983924: Add an OpenClaw plugin for Zuul integration https://review.opendev.org/c/openstack/project-config/+/983924 | 17:28 | |
| @fungicide:matrix.org | you really should | 17:28 |
| @clarkb:matrix.org | does the @ in zuul@openstack.org here: https://lists.openstack.org/archives/list/release-job-failures@lists.openstack.org/thread/DRCXRRA7Y76HO6BGGXFK2C2QJXDI3I54/ render as an ff20 glyph for anyone else? I'm surprised that my font doesn't have the full width commercial at symbol in it | 17:39 |
| @clarkb:matrix.org | and also that mailman translates ascii @ to 0xff20 | 17:39 |
| @clarkb:matrix.org | not a big deal. more of a "huh why would they replace the @ and why is my font broken" | 17:39 |
| @mordred:waterwanders.com | Clark: it looks normal to me? | 17:40 |
| @mordred:waterwanders.com | although - I wonder if it's an attempt to stop email scrapers? | 17:40 |
| @clarkb:matrix.org | oohhh yes I bet that is the purpose | 17:41 |
| @fungicide:matrix.org | yes, i think hyperkitty may do that translation | 17:41 |
| @clarkb:matrix.org | my three fonts are serif: liberation serif, sans-serif: roboto, and monospace: source code pro | 17:42 |
| @clarkb:matrix.org | that looks sans serif so roboto must not include ff20? | 17:42 |
| @fungicide:matrix.org | my browser's fontset renders that glyph visually similar enough tio @ as to be indistinct | 17:42 |
| @clarkb:matrix.org | or maybe it is monospace and it is source code pro without the code point | 17:42 |
| @fungicide:matrix.org | page source also uses zuul＠openstack.org so it's done in the content | 17:43 |
| @fungicide:matrix.org | my terminal renders it in cjk/east asian width, but my browser must be kerning it to approximately normal western width | 17:45 |
| @clarkb:matrix.org | https://fonts.google.com/specimen/Roboto?specimen.preview.text=%EF%BC%A0 and https://fonts.google.com/specimen/Source+Code+Pro?specimen.preview.text=%EF%BC%A0 both imply maybe neither have it. Except i don't know if the ff20 char is being explicitly used there rather than the codepoint | 17:47 |
| @clarkb:matrix.org | it renders on my phone which should also use roboto so probably something wtiht my system fonts or firefox fonts. Weird | 17:51 |
| @fungicide:matrix.org | https://gitlab.com/mailman/hyperkitty/-/commit/133172b | 17:51 |
| @fungicide:matrix.org | classic commit message | 17:52 |
| @fungicide:matrix.org | but it's been doing that since the mm2 days, commit authored almost 14 years ago | 17:52 |
| @fungicide:matrix.org | though hyperkitty, not pipermail (which is what we used for the mm2 archive frontend) | 17:53 |
| @fungicide:matrix.org | or maybe that was for very early mm3, but was back in the python 2.x version | 17:53 |
| @fungicide:matrix.org | anyway, no related explanation that i can find, so mordred's guess matches mine | 17:54 |
| @fungicide:matrix.org | it's at least been that way for as long as we've been using hyperkitty, from what i can tell | 17:55 |
| @clarkb:matrix.org | ya that explanation makes sense to me | 17:57 |
| @clarkb:matrix.org | I did some testing in a terminal and none of my monospcae fonts there seem to know what to do with the character. Not a huge deal | 17:57 |
| @clarkb:matrix.org | it does plenty of other unicode just fine including 4 byte chars so its weird that a 2 byte would be missing | 17:58 |
| @clarkb:matrix.org | https://ubuntu.social/@launchpadstatus has more lp info though doesn't have notes about the most recent issues. It does call out git as being a problem elsewhere. Git really is a trap for the crawlers. | 18:04 |
| @fungicide:matrix.org | 99.99999% (or more) of the web is not git-oriented content, so it's understandable that typical web crawlers don't optimize for it | 18:05 |
| @harbott.osism.tech:regio.chat | also maybe 99% of the git-web content is github, so easy to just special case that and ignore everyone else? | 18:24 |
| -@gerrit:opendev.org- Jeremy Stanley https://matrix.to/#/@fungicide:matrix.org proposed: [opendev/system-config] 984123: Refresh the Puppetlabs Secure APT repository key https://review.opendev.org/c/opendev/system-config/+/984123 | 20:35 | |
| @fungicide:matrix.org | that ^ turned out to be simpler than expected, just a bit of legwork to track down and test | 20:35 |
| @fungicide:matrix.org | i do fully expect other errors once that's fixed, but will deal with them in additional changes | 20:46 |
| @clarkb:matrix.org | I +2'd it | 21:01 |
| @clarkb:matrix.org | I did not grab the key and check it independently. Mostly trusting that fungi did that leg work | 21:01 |
| @clarkb:matrix.org | also last call on whether or not anything needs doing before Sunday's Gerrit upgrade. I'm about to do a school run and am plannnig to unplug after that | 21:01 |
| @fungicide:matrix.org | i did read back through the etherpad and i'm good with what's there. planning to be around starting at 19:00 utc to start prep steps ahead of the outage | 21:12 |
| @clarkb:matrix.org | See you then | 21:16 |
| @fungicide:matrix.org | likewise! | 21:17 |
| @fungicide:matrix.org | https://lists.openstack.org/archives/list/openstack-discuss@lists.openstack.org/message/G5IVTDP2TNGG6DNN6CQOVZESJFN6VZ7J/ proposes removing nova-network support from osc, will that impact our use of rackspace classic? | 21:48 |
| @clarkb:matrix.org | Unclear. We don't explicitly manage networks in tax classic but I don't know if that relies on support within osc | 21:49 |
| @clarkb:matrix.org | Maybe we can try a build with support removed and see if it works | 21:49 |
| @fungicide:matrix.org | oh, good point, if we don't need to add/modify networks then it's likely irrelevant | 21:49 |
| @fungicide:matrix.org | Clark: how far ahead did you want to un-wip and merge 983474? | 21:56 |
| @fungicide:matrix.org | on sunday, or sooner? | 21:56 |
| @fungicide:matrix.org | i assume we're planning to do steps 4-6 in the plan about an hour ahead at 19:00 utc | 21:57 |
| @clarkb:matrix.org | fungi: I think for 983474 we unwip and approve it once we're running on gerrit 3.12. That is step 30 | 22:02 |
| @clarkb:matrix.org | and yes 4-6 should happen around 1900 | 22:02 |
| @clarkb:matrix.org | step 3 is mostly just to remind us to be prepared for step 30 | 22:02 |
| @clarkb:matrix.org | write and push the change and make sure it is mergeable | 22:02 |
| @clarkb:matrix.org | but don't merge it until later in step 30 | 22:02 |
| @fungicide:matrix.org | ah okay | 22:08 |
| @fungicide:matrix.org | so that step is done already then | 22:08 |
| @fungicide:matrix.org | i'll mark it off since i've +2'd the wip change | 22:08 |
| @clarkb:matrix.org | Sounds good thanks | 22:11 |
Generated by irclog2html.py 4.1.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!