Writing correct software, especially at a lower level, is like 10x harder than people think. It’s always been hard, and now llms can actually fix the obscure concurrency and memory safety bugs. I’m all for it.
The problem is LLMs always sound very confident when “fixing” things and will very rarely report that it’s out of its depth and refuse to work, and yeah sometimes they genuinely just do good but also unless you’re really vigilant it’s easily possible it’s gonna introduce more bugs than it fixed.
I’m not saying Torvalds is stupid enough to just uncritically let this thing loose on the kernel but I understand the trepidation of open source maintainers about this stuff.
LLMs will actually double down on incorrectness and make even wilder mistakes unless you bully them until they go, “I admit I don’t know how to do it”. Unless a reviewer really knows their stuff they can easily fall victim to trusting a bullshitting LLM.
This is on point. You’ve got to keep pushing back until it realizes some overlooked assumptions and shit like that. I cope by architecting my codebase in a way where I know I have guardrails in place for bad LLM guesses
How do you know you’re not just generating the next token and appending it to your input?
Nah just kidding I kinda tend to agree because techbros are already starting to speak about LLMs like they’re actual conscious beings and it’s getting super weird but saying stuff like “the text predictor that is used as a simulacrum of human reasoning is deficient in such and such ways” is kind of a mouthful and also most non-technical people are gonna stare through you if you talk to them like that.
LLM is a statistical model, it does not reason. It is not capable of neither honesty nor deliberate lies. It works sometimes, because it was trained on thousand standard case analyses by human programmers, but it is also prone to hallucinations, which are unavoidable by design.
I know, the reason I shared that example is because there is no humanizing the LLM and it’s used exclusively for its ability to pretty effectively parse syntax to find common higher level bugs in specific C frameworks.
For context:
I never use LLMs when programming because I find them to be more of a hassle. They show me down and muddy the water on design principles.
Even a “bad” design pattern that you’re consistent with is 1000x better than a “good” design pattern that is inconsistently applied. LLMs are fantastic at the later. Things that look great in isolation, but begin to fall apart when they come up against the weight of a larger project.
Especially when it’s a new project without a ton of existing code for it to kinda copy/paste.
Using one to find minor use after frees that are almost impossible to hit, or duplicated calls is fine by me though. Those are usually a pain to find anyways, and there’s almost always 100 more important things to spend time on. If someone wants to burn tokens creating a backlog of easily fixed, reproducible errors like that all while disclosing exactly what they’re using to find them sure. That’s not annoying at all, especially when they only open the PRs after discussion with the project maintainers.
Well if you’re going to be pedantic, they don’t hallucinate either. They just repeat patterns, and sometimes false patterns can emerge in large data sets.
Well thats not always why they give an incorrect prediction. It’s also because statistics are inherently never going to reflect reality perfectly (“all models are wrong, some are useful”), but also we intentionally inject randomness into LLMs and other ML programs to ensure the output isn’t over fitting and spitting out raw training data. Usually adjustable via seed values, temperature, etc
Writing correct software, especially at a lower level, is like 10x harder than people think. It’s always been hard, and now llms can actually fix the obscure concurrency and memory safety bugs. I’m all for it.
The problem is LLMs always sound very confident when “fixing” things and will very rarely report that it’s out of its depth and refuse to work, and yeah sometimes they genuinely just do good but also unless you’re really vigilant it’s easily possible it’s gonna introduce more bugs than it fixed.
I’m not saying Torvalds is stupid enough to just uncritically let this thing loose on the kernel but I understand the trepidation of open source maintainers about this stuff.
LLMs will actually double down on incorrectness and make even wilder mistakes unless you bully them until they go, “I admit I don’t know how to do it”. Unless a reviewer really knows their stuff they can easily fall victim to trusting a bullshitting LLM.
This is on point. You’ve got to keep pushing back until it realizes some overlooked assumptions and shit like that. I cope by architecting my codebase in a way where I know I have guardrails in place for bad LLM guesses
It doesn’t realise anything. It just generates a single token, appends that to its input, outputs that, ends, and starts again with the new input.
The anthropomorphising muddies discussion.
How do you know you’re not just generating the next token and appending it to your input?
Nah just kidding I kinda tend to agree because techbros are already starting to speak about LLMs like they’re actual conscious beings and it’s getting super weird but saying stuff like “the text predictor that is used as a simulacrum of human reasoning is deficient in such and such ways” is kind of a mouthful and also most non-technical people are gonna stare through you if you talk to them like that.
It’s pretty easy to change “keep pushing back till it realises” to “adjust the prompt to bias outputs towards” and similar.
:feedback loop intensifies:
Lol. Llmao.
Here’s one of the few examples where I’ve seen it be actually used in an okay way.
Full transparency, used to generate a standard check and not do giant thousand line PRs, honesty about false positives, and no bloated claims really.
LLM is a statistical model, it does not reason. It is not capable of neither honesty nor deliberate lies. It works sometimes, because it was trained on thousand standard case analyses by human programmers, but it is also prone to hallucinations, which are unavoidable by design.
I know, the reason I shared that example is because there is no humanizing the LLM and it’s used exclusively for its ability to pretty effectively parse syntax to find common higher level bugs in specific C frameworks.
For context: I never use LLMs when programming because I find them to be more of a hassle. They show me down and muddy the water on design principles.
Even a “bad” design pattern that you’re consistent with is 1000x better than a “good” design pattern that is inconsistently applied. LLMs are fantastic at the later. Things that look great in isolation, but begin to fall apart when they come up against the weight of a larger project.
Especially when it’s a new project without a ton of existing code for it to kinda copy/paste.
Using one to find minor use after frees that are almost impossible to hit, or duplicated calls is fine by me though. Those are usually a pain to find anyways, and there’s almost always 100 more important things to spend time on. If someone wants to burn tokens creating a backlog of easily fixed, reproducible errors like that all while disclosing exactly what they’re using to find them sure. That’s not annoying at all, especially when they only open the PRs after discussion with the project maintainers.
All the code is “in there” – it’s a matter of forcing it to produce correct code. There’s tremendous profit if you can.
The same applies to a random number generator.
Well if you’re going to be pedantic, they don’t hallucinate either. They just repeat patterns, and sometimes false patterns can emerge in large data sets.
Well thats not always why they give an incorrect prediction. It’s also because statistics are inherently never going to reflect reality perfectly (“all models are wrong, some are useful”), but also we intentionally inject randomness into LLMs and other ML programs to ensure the output isn’t over fitting and spitting out raw training data. Usually adjustable via seed values, temperature, etc