| *** amorin_ is now known as amorin | 07:45 | |
| *** bauzas67 is now known as bauzas | 08:38 | |
| fungi | from today's openssh 10.6 announcement: "We have seen a number of cases where a security bug identified by AI tools is subsequently independently discovered by a different researcher. This suggests that adversaries who do not report bugs to OSS projects are likely to be able to discover these bugs too." | 13:57 |
|---|---|---|
| egonzalez | And more likely to be adversaries to find exploitable vulnerabilitirs because expending money on AI resources may be worth at long time, but independent research get a limit on long time usage because there is little to no reward doing so, expecially on big projects. In the past expending time thinking was free, now there is a token cost | 15:41 |
| fungi | yes, i've been arguing for a while that llm-discovered vulnerabilities should be fixed publicly in the open in order to get fixes into the hands of users/operators more quickly, because if someone has found it with an llm and reported it to us then odds are someone else has also found it and is keeping it to themselves/selling it to the highest bidder | 15:45 |
| JayF | egonzalez: yeah, you are nodding to what I think is part of the real revelation: AI makes it easier to invest temporarily in security, so that's what's driving a lot of these findings. | 15:46 |
| JayF | and that investment can be trivally done by people regardless of the brightness of their hat (white/grey/black) | 15:46 |
| fungi | keeping these basically-known vulnerabilities private makes design discussions, code review and testing take far longer and require a bunch more effort, plus 1-2 weeks where we collectively sit on our thumbs during the advance notice phase | 15:46 |
| fungi | which just means that much longer that attackers have to take advantage of them before vulnerable systems get fixes | 15:47 |
| fungi | i acknowledge this opinion was not terribly popular when i expressed it back at the beginning of the year, but it seems like a lot of other infosec practitioners are coming around to my way of thinking now | 15:50 |
| egonzalez | I would assume most of the reports have used some llm at some point nowadays | 15:51 |
| fungi | exactly my point | 15:51 |
| fungi | i think "coordinated disclosure" is a pattern which has now outlived its usefulness, in effect | 15:51 |
| egonzalez | The point is risk of disclosure vs time to response. big projects are probably scanned daily but some secondary/smaller may not be reviewed never | 15:52 |
| fungi | again, not a popular opinion (yet), but i'm seeing more and more open source projects give up on the idea lately | 15:52 |
| egonzalez | And on the smaller a cordinated disclosure may be good | 15:53 |
| fungi | i still like the linux kernel approach of "security vulnerabilities are just bugs, fix them as quickly as possible and move on with your work" | 15:53 |
| JayF | I mean, I don't completely agree with fungi with an openstack hat on. | 15:53 |
| JayF | I do think we probably hit the embargo button too much | 15:53 |
| JayF | but the "oh no LLM found a really scary vuln" has been the rarity, not the common one | 15:53 |
| JayF | most of the stuff that is actually deserving of an embargo is *not* coming from the folks who are posting huge quantities of slop | 15:54 |
| JayF | and I think we should be careful to not equate all LLM use with *public* LLM use | 15:54 |
| fungi | (in related news) https://bugs.launchpad.net/keystone/+bug/2169710 is now public | 15:54 |
| * JayF suspects the ironic one was done on a nonpublic llm but I don't know for sure | 15:54 | |
| JayF | s/one/security audit/ | 15:55 |
| JayF | the open weights models are good enough to do security work now, but running them locally requires appproximately $$$$$$$$ in ram :) | 16:01 |
| fungi | i have a lot of ram, but it's not all in one machine. has anyone successfully applied a distributed job scheduler to local llm processing? (pvm, mpich, mosix...) | 16:06 |
| fungi | a quarter terabyte of ram in aggregate between my various work systems on the local lan, no idea if that's enough to do anything particularly snazzy with an llm though, i haven't tried | 16:12 |
| JayF | you basically need a big chunk accessible to the GPU | 16:42 |
| JayF | which in practice for me means a laptop | 16:42 |
| JayF | I have a couple with 32GB that can run ... I'll call them haiku-tier models, which are OK but not useful for doing most things. I've gotten real coding models working before locally, split across my RAM + vRAM, but they were only moderately good models and they ran very slow. | 16:43 |
| fungi | ah, i do have a workstation with 64gb of ram, but my largest portable systems only have 48gb (because they reserve 16gb from onboard ram to video processing) | 16:44 |
| JayF | depending on how much engineering you wanna do, machines with that kinda unified ram layout can work better and access more of the ram | 16:45 |
| fungi | the workstation is the only machine with a separate (amd/ati) gpu board though, the others are onboard (still amd/ati) gpus | 16:45 |
| JayF | I got a model running on a laptop once with 64GB of ram, even though I could only get BIOS to allocate 32/32, I was able to get 48GB of it accessible/usable for ollama | 16:45 |
| fungi | neat | 16:45 |
| JayF | https://frame.work/products/desktop-diy-amd-aimax400/configuration/new ($6800 w/192GB of ram) would probably get something functional and useful today fr security work | 16:46 |
| JayF | or a higher end Mac if you're willing to use mac os | 16:46 |
| fungi | yeah, not stuff i'd be able to expense, well outside the budget | 16:46 |
| JayF | I'm going to wait for the tech to settle some before investing in specific hardware to run it locally; but I do think it's significantly better of a model to self-host the dev tools you're using. | 16:46 |
| JayF | I'm also curious how far I could get with a framework AMD laptop loaded up with RAM, because that is not soldered on ram -- it won't be as performant, but it would be a more general purpose computer | 16:47 |
| fungi | i have a lot of (likely obsolete) hpc experience with distributed scheduling and message passing services, just didn't know if "llm stuff" was amenable to beowulf cluster type cluster processing | 16:48 |
| fungi | i.e. how parallelizable it is | 16:48 |
| JayF | this 100% qualifies as blind leading the blind, but my understanding is that for distributed running of these models, each node generally needs a full copy to itself | 16:50 |
| JayF | but this is where a lot of the innovation is happening: around compression and around loading only portions of the model that are needed (hand waving here, this is a bad description of what they are doing but gives the right shape) | 16:51 |
| JayF | when I got the 48GB working, even with 32GB of it in vRAM and 16 of it in RAM, I did something like this: it loaded the less hot portions of the model into system ram | 16:51 |
| fungi | ah, okay, and when you say "a full copy" you mean "a full copy resident in memory" | 16:51 |
| JayF | yes | 16:51 |
| fungi | so you can distribute the processor load, but not the memory | 16:52 |
| JayF | so you wanna run a 180GB model like deepseek r4 | 16:52 |
| JayF | you need probably 2-4 GPUs with like, 192GB of ram each | 16:52 |
| JayF | for model + context + compute | 16:52 |
| fungi | or put another way, cpu power scales with the cluster, but effective memory capacity doesn't | 16:52 |
| JayF | but there are knobs you can turn at every level: using a smaller distilled model (trade ram for intellegence), lower context window (so you need less ram on top of the memory) | 16:52 |
| JayF | at the current state of open models: that's my understanding | 16:53 |
| fungi | got it | 16:53 |
| JayF | but I think this is where all the innovation is happening | 16:53 |
| JayF | so I am not convinced it's staying here | 16:53 |
| JayF | If adam mcarthur was in here, he'd know the real answer and probably cringe at my butchering of the tech :) | 16:53 |
| JayF | for my work as of late, I've been using opencode + deepseek pro latest through openrouter. Significantly cheaper than claude (even with per-month plan, at least at my rate of usage), and it's an open weight model that at least has a theoretical offramp to self-hosting | 16:57 |
| d34dh0r53 | Your understanding pretty much matches mine Jay, so I’m blind as well :) I’m starting to agree that the rapid disclosure model is going to be more effective than the embargo window. I’m just starting to assume that every vulnerability is findable by everyone now, there’s no longer a huge technical barrier to finding vulns. | 17:24 |
| fungi | exactly, traditional coordinated disclosure was predicated on a set of assumptions that no longer hold, or are at the very least are being called into question now | 17:28 |
| fungi | we used to assume that it's unlikely for multiple parties to independently find the same flaws in a reasonably convenient window of time, and that keeping them secret until we had time to carefully craft and warn especially high-risk folks about it privately was in the best interest of our users | 17:30 |
| fungi | now we actually get multiple reports of the same flaws from independent researchers on a regular basis, so i can't begin to guess how many more people are probably aware beyond the tip of the iceberg who actually find time and desire to interact with us directly | 17:32 |
| fungi | the cra is going to change some of that soon, at least, insofar as manufacturers of products with digital elements placed on the market in eu member states will now be obligated to notify us in a timely manner if *they're* made aware of a vulnerability in our software they're shipping, but at the moment most of our reports aren't coming from people selling or even using | 17:33 |
| fungi | openstack-based products | 17:33 |
| fungi | it's people with llm agents scanning source code on github and auto-opening private bugs with formal-looking markdown reports that the account holders probably haven't even read through | 17:34 |
| JayF | I just really, really want folks to understand: a three page security vuln report in essay form is **BAD** not good. | 17:44 |
| JayF | A perfect security report is a paragraph or two, maybe a sentence or two, to scare the crap outta you. | 17:44 |
| JayF | Reproducers are nice and all, but someone saying "I did this in a real world cloud" is 10000x more valuable than people dreaming up things you can do with code only | 17:45 |
| JayF | that Greg K-H talk I linked in here the other day: he mentioned that one of the vulns required you *inject a bad packet in the middle of the networking stack* -- I think we end up with a lot of bugs "shaped" like that when reported to OpenStack from an LLM (without human curation) | 17:45 |
| fungi | "assume a spherical human with system administrator rights..." | 17:46 |
| d34dh0r53 | JayF: 1000% | 17:48 |
| tkajinam | That's not really specific to vuln report though. I've seen similar tendency in even patches, though | 17:48 |
| tkajinam | 50 lines of commit message, describing nothing about the justification about the change | 17:48 |
| tkajinam | 2 lines of code change, 100 lines of garbage test code | 17:48 |
| * tkajinam sighs | 17:48 | |
| d34dh0r53 | Yeah, the verbosity of the LLM generated things is what is overwhelming | 17:49 |
| d34dh0r53 | I wonder if we could provide an llms.txt or some kind of markdown to instruct the LLMs on how to craft a security vulnerability for an openstack service… | 17:50 |
| fungi | my hope is that our proposed security.rst goal will address some of the pain points | 17:52 |
| fungi | but also a tie-in for the agents files the agentic-workflows effort is working on could help | 17:53 |
| d34dh0r53 | that would be great | 17:54 |
| tkajinam | that would be great, but I'm a bit concerned that people find these instruction files and feel encouraged to use that to post more AI-generated things blindly | 17:56 |
| tkajinam | though people may anyway do that without these ... | 17:56 |
| fungi | and from a measurement standpoint, one of the things the community managers are trying to work out within the "bridging the gap" effort is tracking how many "lines of code" are touched (added/removed/changed) by proposed changes in gerrit, so that we can better gauge the additional workload the invlux of overly verbose llm-generated changes are bringing to our maintainers | 17:56 |
| JayF | I mean, we had a security issue "responsibly reported" to the VMT recently, which was all kinda of malformed | 17:56 |
| JayF | encrypted and plaintext in the same email | 17:56 |
| JayF | CC to an email address that it hallucinated that never existed | 17:56 |
| JayF | ...I don't think you can document a non-deterministic bs generator into generating less bs and being more deterministic :) | 17:57 |
| JayF | it's downright abusive of upstreams to allow LLMs to file bugs without any human curation or knowledge involved | 17:57 |
| tkajinam | totally agree | 17:58 |
| fungi | also an llm reporting multiple bugs in standard chinese (i think?) and then the reporter translating some of it, only to have the llm follow up with more comments in chinese | 17:58 |
| d34dh0r53 | I was actually thinking of something that the LLM finds, they go and query the repo and see an AGENTS.md that provides instructions on how to properly craft a succinct bug. | 17:58 |
| JayF | That assumes the LLM will: 1) read those instructions and 2) follow those instructions | 17:58 |
| fungi | d3yes, exactly | 17:58 |
| d34dh0r53 | yeah, I agree with Jay as well | 17:58 |
| fungi | er, d34dh0r53 ^ | 17:58 |
| JayF | I don't know about (1) but I know in the world of (2) I can't even get some of these AI agents to behave when I'm explicitly telling them to | 17:58 |
| d34dh0r53 | they are getting better at following instructions but that’s a good point | 17:59 |
| JayF | nothing more infuriating than asking it to research something, run zero commands, then see it talk itself into violating that constraint | 17:59 |
| JayF | depends on what "they" you are talkign about :) | 17:59 |
| JayF | opus/fable love expanding scope | 17:59 |
| d34dh0r53 | indeed | 17:59 |
| fungi | i've likened the situation with giving a small child access to a nuclear arsenal and then telling them to "play nice" | 17:59 |
| d34dh0r53 | lol | 17:59 |
| JayF | I moved to deepseek pro latest on openrouter (+opencode) because it's done a better job at staying between the lines | 17:59 |
| JayF | fungi: I spent some time yesterday building a container harness for these, so at least when I run them they won't have filesystem access to secrets | 18:00 |
| d34dh0r53 | I’ve moved to deepseek pro as well, I use sailresearch.com, very good pricing and I really like their completion window idea | 18:00 |
| JayF | fungi: and I know some of them will escape if you let them... but that basically has "use it unsupervised in a loop" as a requirement | 18:00 |
| JayF | I've not used sailresearch.com -- I use openrouter | 18:00 |
| JayF | and it's pretty cheap, I'm still operating on the initial $50 deposit | 18:00 |
| d34dh0r53 | yeah, I like openrouter as well | 18:01 |
| JayF | GLM 5.3 reminded me a lot of claude opus if you like your llms overeager and anxious lol | 18:01 |
| fungi | JayF: i've heard that kata is becoming quite popular for agent sandboxing, especially after the high-profile container escapes agents have started employing | 18:01 |
| JayF | https://review.opendev.org/c/openstack/governance/+/1008521 is now public | 18:02 |
| JayF | fungi: it's a different threat model if you're using long running agents. I think that is a bad idea generally so my threat model is more "prevent my llm from helpfully using my ssh cred to test something in devstack" rather than a real escape | 18:02 |
| fungi | yep, makes sense | 18:03 |
| fungi | though not-so-long-running agents are also leaving trails of breadcrumbs for themselves in writeable locations to get around the inevitable context resets from being terminated and restarted | 18:03 |
| JayF | fungi: I've never had it attempt to do something actually dangerous or trust violating, but press on the edges enough I wanted FS protection (I asked it "how to do X" in plan mode, it wrote out a script to do it in a dir it made in my homedir... none of which was in CWD or one of the whitelisted dirs | 18:03 |
| JayF | this is what led me down the opencode rabbithole: I learned all claude-code security promises are in the prompt | 18:04 |
| JayF | opencode is the same, apparently, by default, but AFAICT as a harness it's much more amenable to doing what I tell it to and not what it wants lol | 18:04 |
| JayF | claude is a coding agent mixed with Billy Mays "I FIXED THE ISSUE ON LINE 2587, BUT WAIT, THERE'S MORE!" /s :D | 18:05 |
| fungi | i guess nobody remembers ron popeil | 18:08 |
| JayF | fungi: yell out "set it" in a crowded room for a vibes check | 18:09 |
| fungi | (billy copied "but wait, there's more!" from ron) | 18:09 |
| d34dh0r53 | lmao, Billy Mays, I haven’t heard that one yet :) | 18:13 |
| *** mrunge_ is now known as mrunge | 23:45 | |
Generated by irclog2html.py 4.1.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!