| -@gerrit:opendev.org- Enzo Candotti proposed: [openstack/project-config] 994994: Add upgrade-operator app to StarlingX https://review.opendev.org/c/openstack/project-config/+/994994 | 00:56 | |
| -@gerrit:opendev.org- Enzo Candotti proposed: [openstack/project-config] 994994: Add upgrade-operator app to StarlingX https://review.opendev.org/c/openstack/project-config/+/994994 | 03:17 | |
| -@gerrit:opendev.org- Benedikt Löffler proposed: [zuul/zuul-jobs] 995029: ensure-python: fix stow env match for python version https://review.opendev.org/c/zuul/zuul-jobs/+/995029 | 11:18 | |
| -@gerrit:opendev.org- Michal Nasiadka proposed wip on behalf of Mohammed Naser: [opendev/system-config] 980994: Add node_exporter role for stats reporting https://review.opendev.org/c/opendev/system-config/+/980994 | 12:23 | |
| -@gerrit:opendev.org- Michal Nasiadka proposed wip on behalf of Mohammed Naser: [opendev/system-config] 980994: Add node_exporter role for stats reporting https://review.opendev.org/c/opendev/system-config/+/980994 | 12:25 | |
| -@gerrit:opendev.org- Clark Boylan proposed: [opendev/system-config] 994565: Make public ip address public not private https://review.opendev.org/c/opendev/system-config/+/994565 | 12:29 | |
| -@gerrit:opendev.org- Michal Nasiadka proposed wip on behalf of Mohammed Naser: [opendev/system-config] 980994: Add node_exporter role for stats reporting https://review.opendev.org/c/opendev/system-config/+/980994 | 12:29 | |
| -@gerrit:opendev.org- Michal Nasiadka proposed wip on behalf of Mohammed Naser: [opendev/system-config] 980994: Add node_exporter role for stats reporting https://review.opendev.org/c/opendev/system-config/+/980994 | 12:31 | |
| -@gerrit:opendev.org- Michal Nasiadka proposed wip on behalf of Mohammed Naser: [opendev/system-config] 980994: Add node_exporter role for stats reporting https://review.opendev.org/c/opendev/system-config/+/980994 | 12:35 | |
| -@gerrit:opendev.org- Michal Nasiadka proposed: [opendev/system-config] 988310: Add GrepTimeDB long term storage for Prometheus https://review.opendev.org/c/opendev/system-config/+/988310 | 12:35 | |
| -@gerrit:opendev.org- mnasiadka.admin marked as active on behalf of Mohammed Naser: [opendev/system-config] 980994: Add node_exporter role for stats reporting https://review.opendev.org/c/opendev/system-config/+/980994 | 12:40 | |
| -@gerrit:opendev.org- Michal Nasiadka proposed on behalf of Mohammed Naser: [opendev/system-config] 980994: Add node_exporter role for stats reporting https://review.opendev.org/c/opendev/system-config/+/980994 | 12:42 | |
| -@gerrit:opendev.org- Michal Nasiadka proposed on behalf of Mohammed Naser: [opendev/system-config] 980994: Add node_exporter role for stats reporting https://review.opendev.org/c/opendev/system-config/+/980994 | 12:45 | |
| -@gerrit:opendev.org- Michal Nasiadka proposed: [opendev/system-config] 988310: Add GrepTimeDB long term storage for Prometheus https://review.opendev.org/c/opendev/system-config/+/988310 | 12:46 | |
| @jim:acmegating.com | > <@mordred:waterwanders.com> corvus: mostly mentioning in case there's a real problem - but I pushed up a new version of 989910 (patchset 9) at 15:25 - so almost 20 minutes ago - and zuul status is still showing jobs in progress for patchset 8 | 14:27 |
|---|---|---|
| `2026-06-25 22:25:25,951 ERROR zuul.GerritEventProcessor: requests.exceptions.ConnectionError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))` | ||
| @jim:acmegating.com | looks like the ai crawlers that were smothering gerrit yesterday were effective enough to affect zuul | 14:28 |
| @clarkb:matrix.org | how fun | 14:29 |
| @jim:acmegating.com | that happened 16 times yesterday | 14:29 |
| @jim:acmegating.com | 26 the day before | 14:30 |
| @jim:acmegating.com | 3 the day before | 14:30 |
| @mordred:waterwanders.com | wow | 14:32 |
| @clarkb:matrix.org | I think https://review.opendev.org/c/opendev/system-config/+/994798 is a next step for trying to make that better | 14:32 |
| @mordred:waterwanders.com | what are the AI crawlers _doing_ to gerrit? | 14:32 |
| @mordred:waterwanders.com | like - gitea I get | 14:32 |
| @clarkb:matrix.org | That won't fix things when we're truly overwhelmed but I think it will give us more headroom when things are generally busy but keeping up | 14:33 |
| @clarkb:matrix.org | mordred: they are crawling the APIs now | 14:33 |
| @mordred:waterwanders.com | ah. clever | 14:33 |
| @mordred:waterwanders.com | I feel like we're being chased by velociraptors | 14:33 |
| @clarkb:matrix.org | I was looking through Anubis' precanned ruleset stuff yesterday and its quite eye opening | 14:34 |
| @clarkb:matrix.org | they've done quite a bit of work to classify the crawlers and their origins and behaviors and have rules setup to identify them. They are also running a paid service that aggregates global data from subscribers into a central reputation db so that you can get the cloudflare effect if running anubis | 14:35 |
| @clarkb:matrix.org | but yes tl;dr is the internet is dead | 14:35 |
| @clarkb:matrix.org | Its to the point where I wonder if we need to put everything behind wireguard and if you want to contribute you need to send an email with your public key to get added because amazingly email is less bad at this point | 14:35 |
| @fungicide:matrix.org | oh, i thought we had merged 994798 already | 14:37 |
| @fungicide:matrix.org | held etherpad 3.3.2 node at 50.56.159.16 is working for me, so approving 994325 | 14:58 |
| @clarkb:matrix.org | thanks I'm around to help monitor | 15:02 |
| @clarkb:matrix.org | fungi: did you want to respond to stephen about identity v2 as well? | 15:48 |
| @clarkb:matrix.org | I think we may rely on that somewhere too though our clouds.yaml doesn't immediately make it obvious if os | 15:49 |
| @fungicide:matrix.org | i do want to respond to him, it remains to be seen whether i'll find the time | 15:52 |
| -@gerrit:opendev.org- Clark Boylan proposed: [opendev/system-config] 995096: Bump Anubis difficulty to 5 https://review.opendev.org/c/opendev/system-config/+/995096 | 15:54 | |
| @jim:acmegating.com | Clark: i approved 798 is that okay? | 16:15 |
| @jim:acmegating.com | i saw "i'm around to help monitor", hit approve, then realized you were talking about etherpad | 16:15 |
| @jim:acmegating.com | but i suspect it may be true for gerrit as well, so i'm leaving my +3 unless you tell me stop | 16:16 |
| @clarkb:matrix.org | corvus: yes true for both | 16:16 |
| @clarkb:matrix.org | looks like etherpad failed on some timeout? | 16:17 |
| @clarkb:matrix.org | I will recheck it | 16:17 |
| @jim:acmegating.com | 2026-06-26 15:11:52.473844 | bridge99.opendev.org | RUNNING HANDLER [base/repos : Update apt cache] ******************************** | 16:18 |
| @jim:acmegating.com | that's the lang thing it did, so... i assume ai crawlers killing ubuntu archives or something | 16:18 |
| @jim:acmegating.com | * that's the last thing it did, so... i assume ai crawlers killing ubuntu archives or something | 16:18 |
| @fungicide:matrix.org | probably | 16:19 |
| @clarkb:matrix.org | :/ | 16:19 |
| @mnasiadka:matrix.org | Are we back to Clark’s idea with wireguard? :-) | 16:54 |
| @fungicide:matrix.org | will we be able to wg to ubuntu package repositories? ;) | 17:19 |
| @fungicide:matrix.org | looks like the gerrit change is failing in the gate | 17:22 |
| @clarkb:matrix.org | the gerrit issue is with quay errors | 17:22 |
| @fungicide:matrix.org | "504 Gateway Time-out" pulling the mariadb container image, yep | 17:23 |
| @fungicide:matrix.org | i'm going to need to disappear shortly to run some pre-travel errands | 17:24 |
| @fungicide:matrix.org | looks like the etherpad upgrade should merge at least, but with quay acting up it still may break trying to deploy | 17:26 |
| @mnasiadka:matrix.org | fungi: well, we have Ubuntu mirrors, but not Quay mirrors ;) | 17:27 |
| @fungicide:matrix.org | rather, we use quay *as* a mirror | 17:28 |
| @mordred:waterwanders.com | gotta love when your mirrors need mirrors | 17:32 |
| -@gerrit:opendev.org- Zuul merged on behalf of Clark Boylan: [opendev/system-config] 994325: Update to Etherpad 3.3.2 https://review.opendev.org/c/opendev/system-config/+/994325 | 17:35 | |
| @fungicide:matrix.org | oh that merged faster than i expected | 17:35 |
| @fungicide:matrix.org | in deploy well agead of hourly jobs | 17:36 |
| @fungicide:matrix.org | ahead | 17:36 |
| @fungicide:matrix.org | i really need to head out now though. hopefully i'm not leaving folks in the lurch on testing the etherpad upgrade deployment | 17:37 |
| @clarkb:matrix.org | I'm following it | 17:39 |
| @clarkb:matrix.org | its up and healthy now according to docker | 17:42 |
| @clarkb:matrix.org | https://etherpad.opendev.org/p/gerrit-upgrade-3.13 loads for me | 17:43 |
| @mordred:waterwanders.com | lgtm | 17:56 |
| -@gerrit:opendev.org- Adam Harwell proposed: [openstack/project-config] 995124: Add x/devstack-plugin-asgi repository https://review.opendev.org/c/openstack/project-config/+/995124 | 18:27 | |
| @rm_work:matrix.org | ^^ I am trying to do some research / proof of concept work on what ASGI could look like -- would it be realistic to get this x/ repository to make that work streamlined a bit, or should I just put the devstack work copy/pasted and embedded in all of my WIP testing CRs? | 18:37 |
| @rm_work:matrix.org | that seemed to be the pattern people use | 18:38 |
| @rm_work:matrix.org | also, hey all, nice to be back in OpenStack :D I am obviously feeling ... possibly over-ambitious ;) | 18:38 |
| @rm_work:matrix.org | I don't suppose anyone here knows if there is actually a SIG / effort already existing for ASGI research | 18:41 |
| @rm_work:matrix.org | but I am willing to kind of... spearhead this :D as I got Hypercorn added to g-r last week | 18:42 |
| @clarkb:matrix.org | if you aren't committing to it long term and an eventually result is maybe going to end up in devstack anyway then yes pushing a test change is fine | 18:45 |
| @rm_work:matrix.org | yeah | 18:45 |
| @rm_work:matrix.org | would be happy for it to eventually land in devstack prime] | 18:45 |
| @rm_work:matrix.org | and if my experiment does not work out or the result is "we looked at this stuff and want a SIG to do a real plan" I am happy to throw it all away and go with that lol | 18:46 |
| @rm_work:matrix.org | very low stress, but I FEEL like I can probably... get something that looks good just ... done and show it off as an option | 18:46 |
| @rm_work:matrix.org | alternatively I can just throw the plugin in some central place temporarily and Depends-On that I think even if it doesn't really belong there, like oslo.service since I will likely be Depends-On that in every change anyway? it'll just look gross but technically it should work | 18:48 |
| @rm_work:matrix.org | ah do we not use x/ namespace anymore? | 18:51 |
| @rm_work:matrix.org | a lot of my upstream knowledge is like 5 years up to date T_T | 18:52 |
| @rm_work:matrix.org | *out of date | 18:52 |
| @fungicide:matrix.org | okay, back for a bit | 18:52 |
| @clarkb:matrix.org | rm_work: we do not for new stuff | 18:52 |
| @mnasiadka:matrix.org | Yes, x/ is not something anybody wants to add to | 18:52 |
| @fungicide:matrix.org | etherpad post-update is still working great for me | 18:52 |
| @clarkb:matrix.org | which is why I was pushign towards just udpating devstack and testing that way | 18:53 |
| @clarkb:matrix.org | adding another throwaway repo to the graveyard is innecessary work | 18:53 |
| @rm_work:matrix.org | ok so I should just make CRs to devstack itself and Depends-On | 18:53 |
| @rm_work:matrix.org | that's fine with me :) | 18:53 |
| @fungicide:matrix.org | rm_work: when adding new projects the recommendation is to use a relevant namespace name, picking a wholly new namespace if there's not an appropriate one | 18:54 |
| @mnasiadka:matrix.org | rm_work: I think maybe it’s fine the experiment lives in a stack of changes in devstack and you can then summarise your experiences in a mail to openstack-discuss? | 18:54 |
| @fungicide:matrix.org | but yeah, working within an existing project is even better still | 18:54 |
| @rm_work:matrix.org | cool yeah I am happy for lowest friction | 18:54 |
| @rm_work:matrix.org | I just looked at the model of existing devstack plugins, and then assumed x/ would be the place, but obviously that's my old reflexes | 18:55 |
| @rm_work:matrix.org | so I'll just make a CR stack in devstack repo :) | 18:55 |
| @mnasiadka:matrix.org | rm_work: it probably would make more sense to get some feedback from gmann on #openstack-qa on IRC | 18:55 |
| @rm_work:matrix.org | yeah I just wanted to actually get some trial runs in so I know enough about the real issues to talk intelligently about it / point at real code for reference | 18:56 |
| @mnasiadka:matrix.org | We’re not in a position to decide for devstack :) | 18:56 |
| @rm_work:matrix.org | obviously it seems like the kind of big cross-project effort that deserves discussion | 18:56 |
| @rm_work:matrix.org | but discussion is always easier if you can point to code and say "this works (or doesn't) because XYZ" | 18:57 |
| @rm_work:matrix.org | maybe folks say "wow yeah that looks amazing" and we could just use it mostly, or maybe it just illustrates a possibility and people have great other ideas about how to do it more cleanly and I abandon it all | 18:57 |
| @rm_work:matrix.org | but yeah if I am not using a temp repo then it isn't really a discussion for here anyway, other than ya'll have a good pulse on stuff that's going on so if anyone knows why what I'm doing is dumb up-front, please let me know ;) | 18:58 |
| @fungicide:matrix.org | also good to see you around again, it's been a while! | 19:00 |
| @rm_work:matrix.org | yeah I have missed it <3 | 19:00 |
| @fungicide:matrix.org | it's always wonderful when old friends come back to us | 19:00 |
| @rm_work:matrix.org | with AI stuff I have been able to get enough internal work done quickly enough that I actually have some breathing room to poke my head back upstream :D | 19:00 |
| @clarkb:matrix.org | rm_work: I think one of the disconnects people have working with Gerrit coming from Github is that people think you need a proper repo because everything happens ina fork on github | 19:09 |
| @clarkb:matrix.org | with Gerrit creating repos is expensive but pushing refs for review is dirt cheap | 19:09 |
| @rm_work:matrix.org | yeah I am used to depends-on but I figured since most devstack plugins live in repos that I would follow the model | 19:09 |
| @clarkb:matrix.org | and the tools like zuul work well with that setup | 19:09 |
| @rm_work:matrix.org | but I guess that's only older stuff | 19:09 |
| @clarkb:matrix.org | I think its more for stuff that devstack isn't maintaining themselves | 19:09 |
| @clarkb:matrix.org | so like you want ceph that is separate. But devstack manages wsgi directly. It should rpobably manage asgi directly too | 19:10 |
| @rm_work:matrix.org | yeah like the devstack-kafka and devstack-container stuff is separate repos | 19:10 |
| @clarkb:matrix.org | basically the plugins are for non default special cases. Which asgi might be today but maybe isn't in the future? Idunno | 19:10 |
| @clarkb:matrix.org | there was a group trying to figure that out for openstack and I'm not sure where it ended up | 19:10 |
| @clarkb:matrix.org | infra-root I rechecked https://review.opendev.org/c/opendev/system-config/+/994798 as quay status doesn't seem to report current issues | 19:11 |
| @rm_work:matrix.org | yeah anyway I am also fine with being told it doesn't belong in devstack, but for my testing they can't keep me from having a CR to depends-on, the most that can happen is a -2 :P | 19:12 |
| @fungicide:matrix.org | also it's a good way to 1. get a head start on the poc where everyone can weigh in, and 2. figure out if it does indeed warrant a separate plugin repo | 19:13 |
| @rm_work:matrix.org | yep | 19:15 |
| @rm_work:matrix.org | I sometimes feel the need to make it clear that I have very little ego attached to any of the stuff I'm doing, and with AI heavily involved even less so, I will throw away any of this in the blink of an eye if someone proposes something better lol | 19:16 |
| @mordred:waterwanders.com | fungi: if you have a sec, https://review.opendev.org/c/openstack/pbr/+/989390 is green. you reviewed it yesterday, but I had to add a different py2 version exclusion due to the way easy_install and the venv-based tests interact, so worth a quick relook | 19:39 |
| @fungicide:matrix.org | mordred: that's the new `pbr(dist)` function defined in `pbr/__init__.py` i guess? | 19:50 |
| @fungicide:matrix.org | and removal of the other condition from `pbr/pyprojecttoml.py` | 19:51 |
| @fungicide:matrix.org | lgtm, thanks! | 19:51 |
| @fungicide:matrix.org | disappearing again for a bit | 19:52 |
| @mordred:waterwanders.com | thank you!!! | 19:56 |
| @mnasiadka:matrix.org | Clark: it seems the rebased node exporter patch feels fine with your changes - I’ll be back at 7th July, so if you feel like pushing the stack forward - that could unblock some follow up work | 20:47 |
| @clarkb:matrix.org | mnasiadka: ack thanks for checking. I think that only affects testing so it should be safe to proceed and revert if it comes to that. | 20:52 |
| @clarkb:matrix.org | The Gerrit apache update is back in the gate finally | 20:52 |
| @fungicide:matrix.org | oh good | 21:08 |
| -@gerrit:opendev.org- Zuul merged on behalf of Clark Boylan: [opendev/system-config] 994798: Increase the Gerrit apache worker limits https://review.opendev.org/c/opendev/system-config/+/994798 | 21:24 | |
| @clarkb:matrix.org | that was quick | 21:26 |
| @clarkb:matrix.org | I see the configuration is in place and apache was restarted. I can still reach Gerrit's web ui | 21:28 |
| @clarkb:matrix.org | fungi: not sure that you're still around at this point but if you want to you could check that the status dashboard reports more slots now | 21:28 |
| @clarkb:matrix.org | but I'm reasonably happy this didn't break anything upfront | 21:29 |
| @fungicide:matrix.org | yep, looks bigger now: https://paste.opendev.org/show/bHZowihNtX4lrFxX0Mnr/ | 22:37 |
| @fungicide:matrix.org | also not at all busy at the moment, apparently | 22:39 |
| @fungicide:matrix.org | `R` is reading a request, `W` is writing a response, `_` is a waiting worker slot available for a request, and `.` is unused capacity because the worker processed were spun down completely and will be handled by newly forked processes when request volume picks back up again | 22:41 |
| @fungicide:matrix.org | s/processed/processes/ | 22:41 |
| @mordred:waterwanders.com | huzzah! the pbr stack up through adding resolute and py315 testing has all landed \o/ | 22:47 |
| @fungicide:matrix.org | awesome. i don't know if next week is great timing for prerelease testing as i won't be around, but if others want to attempt it i look forward to seeing the outcome | 22:49 |
| @clarkb:matrix.org | we'll see how my week goes maybe | 23:40 |
Generated by irclog2html.py 4.1.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!