| @harbott.osism.tech:regio.chat | infra-root: regarding the "AFS is at 95%" topic: is there any hard blocker to simply adding yet another volume to the affected servers? obviously we'd still need to consider pros and cons, but I'm not convinced that the "we need to reduce usage" action item is a strict given | 08:14 |
|---|---|---|
| @harbott.osism.tech:regio.chat | also if we really decide to drop arm64 repo mirrors, I wonder whether we might consider dropping most (or all) arm64 testing together with it, might depend on what impact this will have on job performance. maybe we'd want to run some test jobs without mirrors configured first to evaluate? | 08:17 |
| -@gerrit:opendev.org- Zuul merged on behalf of Michal Nasiadka: [opendev/system-config] 1008055: reprepro: Disable Contents generation for remaining repos https://review.opendev.org/c/opendev/system-config/+/1008055 | 08:56 | |
| @mnasiadka:matrix.org | Ok then, cleaned up all Contents-amd64.gz and did vos releases, should be good now | 09:34 |
| @mnasiadka:matrix.org | Jens Harbott: I think the answer to your question is here: https://etherpad.opendev.org/p/opendev-preptg-202609#L71 | 09:44 |
| @mnasiadka:matrix.org | Probably an option would be to have more than only one partition, but with 95% usage we're currently risking a vos release that would write a lot of data (e.g. some Debian/Ubuntu point release) | 09:45 |
| @amusso:matrix.org | Clark: thank you for the fix of whatever got broken yesterday, that got my change in after a few `reverify` | 12:23 |
| @fungicide:matrix.org | Jens Harbott: in the near term, we hope dropping source packages from the debian mirrors will buy us some time | 13:46 |
| @fungicide:matrix.org | but yes, the concern with adding more volumes is that it increases the odds of a volume failure taking the entire server down followed by someone spending days babysitting the rebuild/recovery/resync | 13:47 |
| @clarkb:matrix.org | Right that is my primary concern. But I also think it is worth evaluating the value of these mirrors. If we run a few hundred arm jobs a day does that justify 2x the mirror cost? Maybe not. We can probably run a few hundred arm jobs a day without terabytes of extra storage. I don't think we have to remove arm completely. | 14:37 |
| But we should see what the Debian source package cleanup gets us and go from there | ||
| @jim:acmegating.com | more partitions would likely reduce the blast radius of a storage failure; would still need to release big volumes but maybe not as many | 14:44 |
| @jim:acmegating.com | https://wikimediafoundation.org/news/2026/10/05/openai-rogue-agent-activities-found-on-wikimedia-projects/ | 14:48 |
| @jim:acmegating.com | apparently openai tried to hack wikimedia's etherpad instance | 14:48 |
| @clarkb:matrix.org | https://review.opendev.org/c/opendev/system-config/+/1008520 and https://review.opendev.org/c/opendev/system-config/+/1008547 would be the next afs steps and should be non impactful if anyone else wants to take a quick look at those (first cleans up unused mirror management stuff and second is the debian source package removal) | 14:58 |
| @clarkb:matrix.org | > <@jim:acmegating.com> apparently openai tried to hack wikimedia's etherpad instance | 15:00 |
| I wonder what the path to using it like a proxy would be. Maybe wikimedia runs a citation like plugin in etherpad? | ||
| @fungicide:matrix.org | also random publicly writable services like etherpads and wikis have become critical to agent swarms being able to persist their memories across context resets | 15:06 |
| @clarkb:matrix.org | yup, though wikimedia indicates there doesn't seem to be evidence of that happenign in this case | 15:07 |
| @clarkb:matrix.org | I went ahead and approved the node exporter fix with one +2 since that isn't really in use yet and I want to get that to a point where we can use. Then I also approved the small cleanup of the node exporter grafana dashboard too (that had no reviews but its also just a dashboard, easy to revert and shouldn't directly affect anything) | 16:54 |
| -@gerrit:opendev.org- Zuul merged on behalf of Clark Boylan: [openstack/project-config] 1008493: Drop node exporter dashboard panels for systemd and ntp https://review.opendev.org/c/openstack/project-config/+/1008493 | 17:11 | |
| -@gerrit:opendev.org- Zuul merged on behalf of Clark Boylan: [opendev/system-config] 1008848: Fix node exportor command args and test cases https://review.opendev.org/c/opendev/system-config/+/1008848 | 17:25 | |
| -@gerrit:opendev.org- Clark Boylan proposed: [opendev/system-config] 1009102: Add codesearch nodes to the node exporter group https://review.opendev.org/c/opendev/system-config/+/1009102 | 17:50 | |
| @clarkb:matrix.org | Those two changes that just landed appear to be working well. I'd like to move forward and start adding more nodes and make sure that when node count is >1 everything still works as expected | 17:51 |
| @clarkb:matrix.org | note that codesearch is using a lot of its disk too so a good one to get in :) | 17:51 |
| @mordred:waterwanders.com | fwiw - I've landed a couple of changes to upstream gerrit that improve rendering on mobile devices (like showing the vote columns instead of the patchset size) I *just* missed the initial 3.15 stable branch cut, but I've proposed them as backports. the earlier ui work related to showing hashtags better should be in 3.15 already | 18:10 |
| @clarkb:matrix.org | mordred: did you see the recent message to their mailing list from someone complaining that voting on a touchscreen is difficult due to the bottons being too close together? | 18:12 |
| @mordred:waterwanders.com | (comparison) | 18:13 |
| @mordred:waterwanders.com | oh - hrm, no, I didn't see that. | 18:13 |
| @mordred:waterwanders.com | on a desktop sized touchscreen or a mobile? | 18:13 |
| @mordred:waterwanders.com | (asking because I vote all the time using phone touchscreen and haven't noticed a problem, but I'd be happy to fix it with a point in the right direction) | 18:14 |
| @mordred:waterwanders.com | oh - to _their_ mailinglist. let me go find where I have that routed | 18:14 |
| @clarkb:matrix.org | mordred: https://groups.google.com/g/repo-discuss/c/XQw3qC781zU/m/RnT26Zz-AQAJ | 18:16 |
| @clarkb:matrix.org | looks like linux desktop? And maybe I misread what touch means there | 18:16 |
| @fungicide:matrix.org | oh, so could be touchscreen nav on the desktop ui rather than mobile | 18:17 |
| @mordred:waterwanders.com | I thin kI haven't have that specific problem because the two things I click "W+1 and CR+2" are not vertically adjacent | 18:17 |
| @mordred:waterwanders.com | yeah. still don't have a problem with it on mobile - must be a desktop thing. | 18:18 |
| @fungicide:matrix.org | fwiw i have a lot of touchscreen netbooks with high-res displays and find most desktop-oriented graphical ui layouts suboptimal for use with a finger, or even a stylus sometimes | 18:19 |
| @fungicide:matrix.org | though i don't recall trying to use gerrit's that way | 18:19 |
| @clarkb:matrix.org | I hate how dirty laptop screens get when you actually use the touchscreen on them | 18:22 |
| @clarkb:matrix.org | my kids use computers this way. Its very different to how I would use a computer | 18:23 |
| @fungicide:matrix.org | i find a stylus helps a lot, both to keep the screen from getting messy and to improve control/accuracy. i also have an antistatic glove that only covers the pinky and heel of the hand, if i'm doing serious sketching or something on a screen | 18:25 |
| @mordred:waterwanders.com | my laptop screens get filthy even without trying to use them as a touchscreen | 18:26 |
| @fungicide:matrix.org | i also carry muliple cleaning cloths, like large versions of the ones for eyeglasses, including one sandwiched between the keyboard and screen when i fold the machine up because oils from my fingers transfer from the keys onto the screen otherwise | 18:28 |
| @mordred:waterwanders.com | well - it seems that chromium has an emulated tap function under playwright and that does, in fact, reproduce the issue the person reported. | 18:33 |
| @fungicide:matrix.org | when i decided i needed a third monitor for my workstation a year or two back, i sprang for a 21" displayport-over-usb touchscreen model which has also worked out really well | 18:36 |
| @jim:acmegating.com | speaking of gerrit, i spent some time with Paladox this morning fixing some things in gerrit's zuul system. one of them was that the zuul-results-summary plugin wasn't running jobs because it has a "main" branch but gerrit has "master" so of course branch matching wasn't working. we fixed that with https://gerrit-review.googlesource.com/c/zuul/config/+/639283 | 18:39 |
| @jim:acmegating.com | while working on that, i noticed that davido has been making some changes to that plugin, mostly around keeping the build infrastructure up to date. based on https://gerrit-review.googlesource.com/c/plugins/zuul-results-summary/+/615801 it may currently be broken in the main branch | 18:40 |
| @jim:acmegating.com | that's something to be aware of if we build gerrit images with that. | 18:41 |
| @jim:acmegating.com | (i'm almost certain that claude is implicated in that error) | 18:41 |
| @clarkb:matrix.org | that change is a fix for the plugin? | 18:42 |
| @jim:acmegating.com | those commit messages look like claude-speak; they are almost incomprehensible. | 18:43 |
| @jim:acmegating.com | Clark: that change claims to be a fix for an error introduced in one of the earlier changes: https://gerrit-review.googlesource.com/q/project:plugins/zuul-results-summary | 18:44 |
| @mordred:waterwanders.com | or maybe davido is the original voice claude has been plagarizing! ;) | 18:44 |
| @clarkb:matrix.org | I guess the CI doesn't cover whatever the issue was as the earlier changes all passed too looks like | 18:45 |
| @jim:acmegating.com | i've known davido long enough to know his voice :) | 18:45 |
| @clarkb:matrix.org | fwiw those changes didn't work with stable gerrit which is why I created the stable branches | 18:46 |
| @clarkb:matrix.org | but it wasn't clear to me if the problem was due to api changes or if they were just broken | 18:46 |
| @jim:acmegating.com | yeah, there's no real testing, there never has been. the new job will just run a build, so it's "does it build" testing now. | 18:46 |
| @clarkb:matrix.org | there definitely were api/build changes to deal with | 18:46 |
| @clarkb:matrix.org | all of that will become far more interesting to us after the 3.14 upgrade and we start 3.15 image builds | 18:47 |
| @mordred:waterwanders.com | Clark: https://gerrit-review.googlesource.com/c/gerrit/+/639483 proposed fix for that mailing list issue | 18:51 |
| -@gerrit:opendev.org- Zuul merged on behalf of Clark Boylan: [opendev/system-config] 1008547: Stop mirroring debian source packages https://review.opendev.org/c/opendev/system-config/+/1008547 | 20:16 | |
| -@gerrit:opendev.org- Zuul merged on behalf of Clark Boylan: [openstack/project-config] 1008533: Cleanup afs grafana dashboard wheel graphs https://review.opendev.org/c/openstack/project-config/+/1008533 | 20:19 | |
| -@gerrit:opendev.org- Clark Boylan proposed: [opendev/system-config] 1009126: Upgrade gitea to 28.1.0 https://review.opendev.org/c/opendev/system-config/+/1009126 | 21:04 | |
| @clarkb:matrix.org | 1008547 merged just as the last debian mirroring run completed. So the next run and first run to execute without sources in its list shuold start around 22:00 UTC | 21:13 |
| @clarkb:matrix.org | fungi: I'm poking around storyboard to give it a test run and I wonder if there is a reliable way to get search boxes to work? It seems like if you type text in it is trying to query in the background to give suggestions but half the time if you get ahead of it it will completely reset you back to an empty search box. But also if you do a freeform search you never get results you need to pick out of options from the background fetch (I recall this being the behavior I think). Mostly wondering if there is a way to make this more reliable as a naive user | 21:23 |
| @clarkb:matrix.org | looks like you can just edit urls to get to say opendev/system-config | 21:24 |
| @clarkb:matrix.org | if I start at hte stories search page it seems to be more reliable. But trying this from the landing page was not so happy. I wonder if it needed to warm some caches | 21:28 |
| @clarkb:matrix.org | and then separately does anyone know of a launchpad project that won't mind if I throw a throwaway bug/issue into its bugtracker? It seems we disabled bugs on openstack-ci otherwise I would use that. Nevermind I have one (my old capstone project used lp) | 21:31 |
| @clarkb:matrix.org | ok I may have hit my first problem and I know I remember this coming up in the past but I don't recall the solution nor is one obvious to me: fungi what do you do to create a story when the button to create a story on the pop up is greyed out and not clickable? I have a story title, description and a task title and description. I also picked the project name via the little menu pop up and didn't just type it out. | 21:44 |
| @jim:acmegating.com | drizzle :) | 21:44 |
| @clarkb:matrix.org | drizzle adjacent yes. Drizzle is another good option | 21:45 |
| @clarkb:matrix.org | anyway based on this initial experience I'm already leaning towards lp and I haven't gone over there yet to create an issue. I got the button to work. I have to pick opendev/system-config out of the little options listing before I type out the entire project name | 21:46 |
| @clarkb:matrix.org | so type openstack/syste and then pick opendev/system-config. If you type opendev/system-config then pick opendev/system-config it doesn't work | 21:47 |
| -@gerrit:opendev.org- Zuul merged on behalf of Steve Baker: [openstack/diskimage-builder] 1007800: pip-and-virtualenv don't depend on epel https://review.opendev.org/c/openstack/diskimage-builder/+/1007800 | 21:52 | |
| @clarkb:matrix.org | We would need to use tags to set priorities and low hanging fruit (I think this is ok and gives more flexibility). But I can't seem to get the system to update the tag on my new story... I can type the tag name and have it give me options (I'm trying low-hanging-fruit in this case) and selecting that out of the options doesn't then let me save that value on the story as far as I can tell. I even left a comment hoping it would bundle the two together | 21:56 |
| @clarkb:matrix.org | so ya I'm leaning towards lp. Even if it is less flexible it works in ways that are likely to be more intuitive to users. But I'll go and create an task there too and make sure I'm not going to hit similar problems | 21:56 |
| @clarkb:matrix.org | LP serach is also clunky due to the sheer number of options | 22:02 |
| @clarkb:matrix.org | but it does seem to work and I can set tags and search by tag as well as priority etc | 22:02 |
| @clarkb:matrix.org | I'll sleep on this and give fungi and opportunity to help me understand storyboard before I start using LP but that is my inclination at the moment | 22:02 |
| @clarkb:matrix.org | gitea 28.1.0 screenshots look good at https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_569/openstack/56925ee083a34ab3a1082490ddb036df/bridge99.opendev.org/screenshots/ | 22:05 |
| @jim:acmegating.com | Clark: in storyboard: | 22:09 |
| 1) Type "low" into the tags text entry field | ||
| 2) Click "low-hanging-fruit" completion since it already exists | ||
| 3) Hit *Enter* in the text entry field | ||
| 4) Wait 5 seconds | ||
| 5) Tag should appear added | ||
| @clarkb:matrix.org | aha its the magic enter key that I think I was not doing /me tries | 22:12 |
| @jim:acmegating.com | then you can bookmark: https://storyboard.openstack.org/#!/search?project_id=748&tags=low-hanging-fruit | 22:12 |
| @clarkb:matrix.org | ok ya that worked | 22:12 |
| @clarkb:matrix.org | searching by tag is easier in storyboard than lp. But it doesn't work in the header level search looks like you need a proper search bar? But ya bookmarks likely to be helpful to | 22:13 |
| @mordred:waterwanders.com | Clark: I added launchpad support to spren.bot but haven't done much with it yet. but if you want to interact with launchpad bugs from an agent - I've got you covered :) | 22:32 |
| @mordred:waterwanders.com | I have not yet added storyboard | 22:32 |
| @clarkb:matrix.org | ``` | 22:40 |
| Error: packages database contains unused 'bookworm-backports|main|source' database. | ||
| This usually means you removed some component, architecture or even | ||
| a whole distribution from conf/distributions. | ||
| In that case you most likely want to call reprepro clearvanished to get rid | ||
| of the databases belonging to those removed parts. | ||
| ``` | ||
| @clarkb:matrix.org | fungi: ^ I think that confirms your suspicion we need to manually intervene | 22:40 |
| @clarkb:matrix.org | https://docs.opendev.org/opendev/system-config/latest/reprepro.html#removing-components we've documented the process of cleaning these up | 22:42 |
| @clarkb:matrix.org | I'll go ahead and start on this in a root screen session on mirror-update03 | 22:42 |
| @clarkb:matrix.org | debian-security is done. I don't think it actually cleaned any packages up likely no source packages in the security repo? | 22:50 |
| @clarkb:matrix.org | debian proper is running now. Still some of those steps went quicker than I expected. Makes me wonder if this won't clear up as much space as I hope | 22:54 |
| @clarkb:matrix.org | Once that is done I'll drop my flock locks, but I'll leave the screen session up so that it can be reviewed to see if it did what we want | 22:54 |
| @clarkb:matrix.org | ok thats all done. Flocks are dropped now too (that was relaly fast makes me skeptical that enough data was freed) | 22:55 |
| @clarkb:matrix.org | first two screen windows are where I head the two flock locks and the second pair of windows is where i ran through the documented steps I linked above | 22:57 |
| @clarkb:matrix.org | afsmon should run in a minute and give us updated stats | 22:59 |
| @clarkb:matrix.org | oh its once at hour at :30 not every 30 minutes | 23:02 |
| @clarkb:matrix.org | so another 28 minutes to get afs stats | 23:02 |
| @clarkb:matrix.org | Oh fun dnssec root keys roll on the 11th | 23:05 |
| @clarkb:matrix.org | Its probably going to be fine most everywhere except for a few dark corners | 23:05 |
| @clarkb:matrix.org | ok afsmon has run and as I feared this doesn't appear to have actually cleared anything out. I'm guessing we weren't properly mirroring source packages or our upstream didn't have them or something? | 23:35 |
| @clarkb:matrix.org | I was relaly hoping this would help but I'm not sure it has (still good hygiene). Time to start thinking about arm64 or other options | 23:35 |
Generated by irclog2html.py 4.1.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!