The problem is LLMs always sound very confident when “fixing” things and will very rarely report that it’s out of its depth and refuse to work, and yeah sometimes they genuinely just do good but also unless you’re really vigilant it’s easily possible it’s gonna introduce more bugs than it fixed.
I’m not saying Torvalds is stupid enough to just uncritically let this thing loose on the kernel but I understand the trepidation of open source maintainers about this stuff.
LLMs will actually double down on incorrectness and make even wilder mistakes unless you bully them until they go, “I admit I don’t know how to do it”. Unless a reviewer really knows their stuff they can easily fall victim to trusting a bullshitting LLM.
This is on point. You’ve got to keep pushing back until it realizes some overlooked assumptions and shit like that. I cope by architecting my codebase in a way where I know I have guardrails in place for bad LLM guesses
How do you know you’re not just generating the next token and appending it to your input?
Nah just kidding I kinda tend to agree because techbros are already starting to speak about LLMs like they’re actual conscious beings and it’s getting super weird but saying stuff like “the text predictor that is used as a simulacrum of human reasoning is deficient in such and such ways” is kind of a mouthful and also most non-technical people are gonna stare through you if you talk to them like that.
The problem is LLMs always sound very confident when “fixing” things and will very rarely report that it’s out of its depth and refuse to work, and yeah sometimes they genuinely just do good but also unless you’re really vigilant it’s easily possible it’s gonna introduce more bugs than it fixed.
I’m not saying Torvalds is stupid enough to just uncritically let this thing loose on the kernel but I understand the trepidation of open source maintainers about this stuff.
LLMs will actually double down on incorrectness and make even wilder mistakes unless you bully them until they go, “I admit I don’t know how to do it”. Unless a reviewer really knows their stuff they can easily fall victim to trusting a bullshitting LLM.
This is on point. You’ve got to keep pushing back until it realizes some overlooked assumptions and shit like that. I cope by architecting my codebase in a way where I know I have guardrails in place for bad LLM guesses
It doesn’t realise anything. It just generates a single token, appends that to its input, outputs that, ends, and starts again with the new input.
The anthropomorphising muddies discussion.
How do you know you’re not just generating the next token and appending it to your input?
Nah just kidding I kinda tend to agree because techbros are already starting to speak about LLMs like they’re actual conscious beings and it’s getting super weird but saying stuff like “the text predictor that is used as a simulacrum of human reasoning is deficient in such and such ways” is kind of a mouthful and also most non-technical people are gonna stare through you if you talk to them like that.
It’s pretty easy to change “keep pushing back till it realises” to “adjust the prompt to bias outputs towards” and similar.
:feedback loop intensifies: