| frickler | config-core: please check https://review.opendev.org/c/openstack/openstack-zuul-jobs/+/979888 | 12:03 |
|---|---|---|
| opendevreview | Dmitriy Rabotyagov proposed openstack/project-config master: Introduce OpenStack-Ansible Power Reviewers group https://review.opendev.org/c/openstack/project-config/+/981924 | 12:03 |
| frickler | clarkb: should we just force-merge https://review.opendev.org/c/openstack/openstack-zuul-jobs/+/965402 or would you want to do further cleanups first? | 12:04 |
| clarkb | I think I'm ok with whatever openstack wants to do to land those cleanups | 13:20 |
| TheJulia | are backends down again? | 13:34 |
| fungi | overloaded with more/different crawlers yes | 13:34 |
| TheJulia | joy | 13:34 |
| fungi | we're staging yet a different mitigation right now | 13:35 |
| TheJulia | Well, at least OI Live is soon :) | 13:35 |
| fungi | yesterday we tuned the haproxy and apache layers to handle a lot more requests so they could reject higher volumes of bogus requests without hitting limits, but now they're making it through to the backend service/database layer so we're at the point where anubis may actually help | 13:36 |
| TheJulia | Would any sort of CDN'ing help? | 13:37 |
| stephenfin | frickler: left some context comments on https://review.opendev.org/c/openstack/openstack-zuul-jobs/+/965402 in favour of force-merging | 13:37 |
| TheJulia | like, actually adjust a budget and pay for something in front a little bit ? | 13:37 |
| fungi | our prior approach of adding obviously faked/unlikely user agent strings to a rejection list has outlived its usefulness i think, now the crawlers are masquerading as current desktop/computer browsers | 13:37 |
| fungi | well, we'd like to not completely give up on the idea that we can run opendev completely on open source software | 13:38 |
| TheJulia | Well, if users are stuck in the mud every time something changes unable to even run tests, at what point do we need to revisit the higher level model | 13:39 |
| fungi | though yes, it would be great if people could convince their employers to pitch in on helping us run these services rather than just complaining about how inconvenient this situation is for them | 13:40 |
| TheJulia | And at some poitn, adding a service is not abandoning base principals when really maybe cache management is just a hard problem | 13:40 |
| TheJulia | i.e. paying someone else running a service to do something you would otherwise burn way more time on | 13:40 |
| TheJulia | just a thought with a businessy hat on | 13:41 |
| fungi | and giving them free reign to track and mine all our user activity in the process, yes | 13:41 |
| TheJulia | There is always a trade-off | 13:41 |
| clarkb | right I thin kthe fundamental issue here is that we have underinvested in opendev and the greater commons for years. We have managed to keep the wheels turning despite this but now we've hit a problem that exposes the cracks | 13:43 |
| clarkb | I personally view "just pay cloudflare to deal with it" as a continuation of this bigger problem | 13:43 |
| clarkb | it may address the immediate frustration but then in 6-24 months yuo're going to hit the next thing that is exposed by this underinvestment | 13:43 |
| fungi | it's too bad nobody who's inconvenienced now cared to invest in making things better before they started falling down around our ears | 13:43 |
| clarkb | and we have been tirelessly working on this for months | 13:44 |
| TheJulia | A multistep plan is really what is needed, buy some time to do the needful. i.e. try to put out the big fire well enough to be able to breath and figure out the right next step | 13:44 |
| clarkb | its not like we've ignored the situiation up until the point it breaks. but I'm expected to do like 8 different jobs in my role which means this particular job only gets a portion of my attention. And that is a similar situation for fungi and others working on opendev. I think this can work if there is enough 1/8 of a role spread across individuals | 13:44 |
| TheJulia | granted, thats hard and there is no magic solution in the end | 13:45 |
| fungi | we do the needful, but thanks | 13:45 |
| clarkb | like do we wnt openstack metrics tracking or do we want gitea to be up? | 13:45 |
| clarkb | at this point its legitimately feeling like people need to start deciding what to axe beacuse I don't have enough hours in the day | 13:45 |
| fungi | it feels like trying to build a fire suppression system in a burning building | 13:51 |
| -opendevstatus- NOTICE: Anubis is now deployed on our Gitea backends, and things are back to working normally though you may notice an Anubis screen flash briefly when starting to browse opendev.org; any jobs which failed prior to 15:00 UTC today can be safely rechecked | 15:34 | |
| opendevreview | Monty Taylor proposed openstack/project-config master: Add gerrit plugin for openclaw https://review.opendev.org/c/openstack/project-config/+/983912 | 16:35 |
| fungi | should i be afraid? ;) | 16:37 |
| fungi | sean-k-mooney: that ^ might also be of interest to you in light of your review bot experiments | 16:39 |
| sean-k-mooney | well i do have an openclaw conencted to matrix... | 16:45 |
| sean-k-mooney | but i use it for almost nothign seince i set it up | 16:45 |
| sean-k-mooney | i was considering haveing it look at patches i have review in teh last 30 days and notify my if any of them have been updated in teh last 24 hours or something | 16:46 |
| fungi | well, i meant more from the perspective of gerrit features supporting ai agents | 16:47 |
| sean-k-mooney | ya ther emight have been a presintation in paris | 16:47 |
| sean-k-mooney | i didnt have time to atend but ti think they had a session on the 3 or 4 attepemt they had to add ai review | 16:47 |
| sean-k-mooney | i guess i need ot look at what the plugin does | 16:48 |
| sean-k-mooney | that woudl eb an alternitive way of triggering it instead of zuul i guess | 16:49 |
| sean-k-mooney | with zuul its has a stonger sandbox then if i just had openclaw do the review as today each review job is an ephmerial k8s pod | 16:50 |
| sean-k-mooney | fungi: but fun to see monty exploring the gerrit ai features | 16:50 |
| sean-k-mooney | fungi: my approch so far to gerrit has been have an agent use a cli to interact | 16:51 |
| sean-k-mooney | i know there is a gerrit mcp too | 16:51 |
| sean-k-mooney | the most usful feaure i have created in my grt tool is `grt comments` which get the commetn form the current review (optionally output in json format) | 16:52 |
| sean-k-mooney | that jsut calls the gerrit rest api but makes it tivial to ask an llm to plan with you how to respond to the feedback | 16:53 |
| * JayF doesn't want those autonomous things anywhere near OpenStack :( | 16:55 | |
| JayF | https://accountable.computer/#UEFSVElDSVBBVEUgSU4gQU4gT1BFTiBTT1VSQ0UgQ09NTVVOSVRZ /me points to the sign | 16:56 |
| sean-k-mooney | i think for openshift they jsut turned on code-rabbit on all the repos | 16:56 |
| sean-k-mooney | JayF: that a refence to an old IBM convetion | 16:57 |
| JayF | sean-k-mooney: I know :) my friend dreid (I think it was dreid?) made it into a webpage where you can swap the last line | 16:57 |
| sean-k-mooney | :) | 16:58 |
| sean-k-mooney | for what its woth my ci job can run on anyting in the openstack namespace | 16:58 |
| sean-k-mooney | but it only runs by default on wathcer/cyborg today | 16:59 |
| JayF | I would have community-shaped problems with any individual who hooked up something like that to an openstack repo without the express consent of some plurality of the community (of that project -- not openstack-as-a-whole) | 16:59 |
| sean-k-mooney | yep that why its only on the those | 17:00 |
| JayF | There are negative network effects around inconsistent LLM-provided code reviews which would increase churn especially for newer contributors :) | 17:00 |
| JayF | I even let claude talk me into giving bad feedback to cardoe yesterday on his change; and I already filtered out 3-4 bad things it had found | 17:00 |
| sean-k-mooney | if you want to try it locally youc an enabel it as a plugin/skill although that is prety new | 17:01 |
| sean-k-mooney | you can also run it by commetinng "teim-ci: manual" | 17:01 |
| JayF | in any project in openstack?! | 17:01 |
| sean-k-mooney | i belive so but i have only tried it in 1 or 2 repos | 17:02 |
| JayF | so the thing I said I'd find problematic has already been done, even if it's manual at this point, and it wasn't even evangelized at any level to the community?! | 17:02 |
| JayF | Is there any public documentation about this? Discussion? | 17:02 |
| sean-k-mooney | https://github.com/SeanMooney/ci-sean-mooney/blob/main/zuul.d/projects.yaml#L9 | 17:03 |
| sean-k-mooney | JayF: no its jsut a third party ci and its intelly not goign to run automaticly on repos where i have not had that dicussion | 17:03 |
| JayF | Code review robots might be third party CI in terms of the CI harness they use, but in terms of community impact it's not the same thing. | 17:04 |
| JayF | Is there a way for me to propose a change banning that third-party CI from ironic-managed repos? So we can at least have that conversation | 17:05 |
| sean-k-mooney | maybe but as i said i dotn gerneally adverise this outside fo converstaion liek this | 17:05 |
| sean-k-mooney | i can change the regex you jus tneed to propsoe a pr. happy to merge it | 17:05 |
| sean-k-mooney | ` ^openstack/.*` can proably be tweaked ot reject anything with ironic in the name | 17:06 |
| sean-k-mooney | or i could jsut comment that out so that i have to leist exptice project | 17:06 |
| sean-k-mooney | like https://github.com/SeanMooney/ci-sean-mooney/blob/main/zuul.d/projects.yaml#L27-L43 | 17:06 |
| JayF | I don't wanna make that decision on behalf of everyone, I want that conversation to happen. And blocking it in a non-openstack repo is just you being nice, not a technical answer to "how to prevent abuse of third party CI to backdoor AI code review" | 17:07 |
| sean-k-mooney | well i guess i wasoul ask is there a polcy that say you cant run a third pary ci in genreal | 17:07 |
| clarkb | the historical answer to "how do we address abuse by third party CI systems" is we disable the gerrit account if project maintainers request it tand as the ci operators to discuss from there | 17:07 |
| sean-k-mooney | we normally have the opicite probelm :) | 17:07 |
| sean-k-mooney | yep that the mecnical/infra solution to abusive account bot or not | 17:08 |
| JayF | I think your characterization of this as a third-party CI is misleading. That is the API interface you're using, but it's not CI in any sense we've used it for. | 17:08 |
| sean-k-mooney | its the same as pep8 form my point of view | 17:09 |
| sean-k-mooney | it just a opinionated linter | 17:09 |
| JayF | pep8 is deterministic, repeatable, and reproducable | 17:10 |
| JayF | as CI is [painful laugh of knowing the real truth] | 17:10 |
| JayF | To be clear: there's no policy I think this is violating, or the like. That's part of why it's so worrisome to me. It's not really an area we've discussed or considered at all at this level | 17:12 |
| sean-k-mooney | ack well i did mention it in the tc channel a few tiems | 17:17 |
| sean-k-mooney | but ya thre hasent been a broader formal dicssion out side of some dicsuson within the watcher team | 17:18 |
| sean-k-mooney | JayF: the actual review system is hosted/defiend here for waht its worth https://github.com/SeanMooney/openstack-ai-style-guide | 17:20 |
| JayF | I have that checked out and use a lot of your files there :) | 17:20 |
| sean-k-mooney | that was orgianlly where i was building a set of md files ot try and have ai conform to openstack coding paractires | 17:21 |
| JayF | my concerns here are completely around community consent and potential to mislead less-senior developers | 17:21 |
| sean-k-mooney | yep i condier moving this to opendev eitehr under the teim-ci name as a new namespace or operoosing it as a openstack project eventually | 17:21 |
| sean-k-mooney | this https://github.com/SeanMooney/openstack-ai-style-guide/blob/master/agents/code-review-agent.md is the guts fo how ti works | 17:22 |
| sean-k-mooney | well https://github.com/SeanMooney/openstack-ai-style-guide/blob/master/agents/teim-review-agent.md and https://github.com/SeanMooney/openstack-ai-style-guide/blob/master/skills/teim-review/SKILL.md are the actul entry point | 17:23 |
| sean-k-mooney | but sure let me update it an im happy to chat about this with the ironic team or any other whenever | 17:24 |
| sean-k-mooney | or if we want to chat about this at a tc level some time that aslo totally fine | 17:24 |
| JayF | I can't/don't wanna speak alone for Ironic, I just mainly don't love the idea that without any single Ironic leader knowing, someone could trigger a gerrit-automated AI review on the change. That seems like a big thing to enable without folks being widely aware. | 17:25 |
| JayF | (replace "Ironic" with other project teams as appropriate) | 17:26 |
| sean-k-mooney | i considerd limiting that to my user orgianlly the only reaso i didnt was i wanted ot ask specific core team member ot try it and give me feedbac | 17:43 |
| sean-k-mooney | JayF: you were going to be on that list by the way but ya let me go tweak it quickly | 17:44 |
| gouthamr | sean-k-mooney: i'd like to facilitate a discussion at the TC / PTG.. i want to gather topics like this one that may need some opinions | 17:44 |
| sean-k-mooney | sure | 17:44 |
| clarkb | fwiw I think experimenting in a small subset of repos with people who are actively involved in maintaining the tool and the software being reviewed is the way to go about this | 17:44 |
| clarkb | regardless of what opinions about utility are if we're going to try anything this is a great wy to do it. I still think itwould be cool to run it in opendev zuul but don't think that is strictly necessary particularly during experimentation | 17:45 |
| gouthamr | folks are using LLMs to do reviews through their own gerrit accounts as well, and i had a few conversations that might be interesting.. i began with the assumption that JayF had - there's a human in the loop, signing off on a review.. but it felt like it wasn't always the case | 17:45 |
| sean-k-mooney | we could espically fi one fo the llm factories were willignto sponsor opensouce usage | 17:46 |
| sean-k-mooney | what i have wont really scale beyound really the scoep fo review that i woudl reveiw | 17:46 |
| JayF | I think one of the community elements to consider is that not every OSS-participating software developer believes the use of LLMs is benefitial and ethical. | 17:47 |
| clarkb | JayF: yes after this week I'm firmly in that camp | 17:47 |
| JayF | Not saying we have to let a vocal minority veto things in the community, but we need to make sure we are leaving space and not forcing those folks out | 17:47 |
| fungi | but avoiding a situation like anthropic's "under cover mode" with claude pretending to be a human on open source projects would be good | 17:47 |
| clarkb | but I also think experimenting and finding what if any value can be utilized is a good idea and its great that sean-k-mooney is able to do it in what appears to be a really hands on approach | 17:47 |
| clarkb | not just a throw it over the wall and good luck but sean-k-mooney is actively reviewing these changes too aiui and can chime in if necessary etc | 17:48 |
| JayF | fungi: I think all contributions to gerrit, both code and review, should be attributed to a human, and that human should be responsible for the tooling they use and compliance with the rules. This is why the CI-based approach doesn't sit well with me. | 17:48 |
| sean-k-mooney | yes i have been trying to go back with my human account and review the review bots output and etierh resovel ti of say yes this is a thing | 17:48 |
| sean-k-mooney | but that also why it wont really scale | 17:49 |
| JayF | clarkb: yeah I'm not worried about sean-k-mooney using this tool; I'm worried about $InternCandidate seeing the comment, repeating it themselves to get the CI feedback, and go deeper down a ad path | 17:49 |
| sean-k-mooney | i have also put a lot of effort into the review critira to make it try and cross check anyting its sayign whcih is why https://github.com/SeanMooney/openstack-ai-style-guide/blob/master/agents/code-review-agent.md is so long | 17:50 |
| clarkb | JayF: yup, but complex systems like this are hard to reason about (both the llm side and the gerrit code review side) in a staging bubble. Eventually you need to try it for real and I think sean-k-mooney has done so responsibly whcih I appreciate | 17:51 |
| JayF | Yeah, I use claude super frequently, I maintain the package for it on Gentoo, I'm not anti-use of the tool; I just think it's similar to an IDE where it needs to be a community<>developer<>LLM flow, not LLM<>community | 17:52 |
| sean-k-mooney | JayF: https://github.com/SeanMooney/ci-sean-mooney/commit/f74ba3daa4e1905c7e32f7988f2c5ef02f44b6c5 :) | 17:53 |
| JayF | sean-k-mooney: danke; although really limiting it to an allowlist of users would be a similarly OK change | 17:53 |
| sean-k-mooney | yep but that requires me to look upt the zuul coment filtering syntax | 17:54 |
| sean-k-mooney | i coudl swap to that later | 17:54 |
| sean-k-mooney | but for now this is fine | 17:54 |
| sean-k-mooney | JayF: gouthamr on a realted topic i was going to bring somethign else to the the mailing list before the ptg | 18:03 |
| sean-k-mooney | ... let me find it | 18:04 |
| sean-k-mooney | i was asked to condier how upstream repos coudl beter be used with ai so in the watcher ptg we will dicuss https://review.opendev.org/c/openstack/watcher-dashboard/+/983336/1 i was basicly in 2 minds git ignore the relevnet files or do that | 18:06 |
| sean-k-mooney | which is comit somethign super minimal and create a space for peopel to experimetn themseve without that every going into the repos | 18:06 |
| sean-k-mooney | the main belive i have however is that if there is a policy or architecture or somethign that woudl be useful for a human or ai to know | 18:07 |
| sean-k-mooney | it shoudl be in teh contrinutor docs | 18:07 |
| sean-k-mooney | and aganst or humans shoudld just read those | 18:07 |
| sean-k-mooney | so any AGENTS.md shoudl basiclly tell teh agent to RTFM | 18:08 |
| fungi | perhaps tangential, but i do find it annoying that executive management at lots of employers (mine included, and those of many of my friends as well) are pushing their employees to "find more uses" for llm technologies | 18:08 |
| fungi | from an engineering perspective, it's called "a solution looking for a problem" | 18:08 |
| fungi | and i'd rather we all just used whatever tools are best suited to solving the problems we have than trying to justify corporate overspending on a particular technology by using it for things it's not adapted to | 18:10 |
| clarkb | https://arxiv.org/html/2408.04667v5 was somethign I found interesting since we'ev talked about determinism a few times | 18:11 |
| clarkb | tl;dr unless you run the model yourself there doesn't seem to be any determinism and even then it requires some care | 18:11 |
| fungi | a friend at oracle (not one of the 30k they just fired on no notice by e-mail last week so they could afford to spend more on ai) says they're all being encouraged to use the internal company chatbot for everything, even though he's noting its answers are wrong somewhere between 20% and 50% of the time | 18:12 |
| sean-k-mooney | redhat hasnt quite gone that far | 18:13 |
| fungi | his theory is that they're trying to train an llm on how to do everyone's jobs | 18:13 |
| sean-k-mooney | but i would not be surpised that other have | 18:13 |
| sean-k-mooney | i mean yes defintely | 18:13 |
| sean-k-mooney | i came acrose this the other day https://www.businesswire.com/news/home/20260407140918/en/WRITER-Survey-Finds-60-of-Companies-Plan-to-Lay-Off-Employees-Who-Wont-Adopt-AI | 18:15 |
| sean-k-mooney | """Employee sabotage: Instead of embracing AI, some workers are pushing back. In fact, 29% of employees — including 44% of Gen Z — admit to sabotaging their company’s AI strategy, for example by entering company information into public tools, using unapproved tools, or refusing to use AI.""" | 18:16 |
| sean-k-mooney | now is not a good time to be in charge of your compaince security/data protection policy adn enfocement | 18:17 |
| fungi | if they were smart, they'd feed it a constant stream of plausible disinformation | 18:17 |
| fungi | and poison the entire model | 18:17 |
| clarkb | ya I mean its not just gitea that they are crawling. Every piece of info you give to claude code is shipped back to base (and I'm sure they all do that, but claude leaked its code so now we know it does | 18:17 |
| fungi | though i guess now that so much web content is llm-supplied hallucinations anyway, that's happening naturally | 18:18 |
| fungi | "model collapse" | 18:18 |
| clarkb | yup, but its also things you think are private like your ssh keys etc (assuming they aren't sufficiently protected from the prying eyes) | 18:19 |
| clarkb | the other news today is that someimtes claude code gives itself permission to do things | 18:21 |
| JayF | fungi: there's an entire prompt engineering discipline that could be summarized as "sabotage the training" | 18:22 |
| opendevreview | Monty Taylor proposed openstack/project-config master: Add an OpenClaw plugin for Zuul integration https://review.opendev.org/c/openstack/project-config/+/983924 | 18:45 |
Generated by irclog2html.py 4.1.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!