Linux is the name of a computer operating system kernel and a group of systems built on it, like Ubuntu or Mint. Linux is the basic infrastructure of the Internet. It’s important. A report ju…
genuine question from me, the uninformed: If a change or commit is being presented as code a person authored (or without indicating one way or another) how can someone tell it is LLM slop vs code a person wrote?
Like, with writing, there are sort of semi-intangible patterns that make me recognize AI slop, but even then I don’t know for sure- its mostly based on suspicion that comes from reading AI slop in the first place. Like one example I can think of is the “Not X, but Y” summarization at the end of nearly any essay or brief explanation of something. Its always meant to be poetic or impactful and it might have been a genuine style at one point -I think it was common in advertisements maybe?- but now it makes me think, “huh, this is probably slop”
So for code, is there some BS that they are always doing that feels inhuman or wrong? Do they make the same errors or add unnecessary shit that makes people roll their eyes? Or is it just a high volume of work with only a short turnover? If someone asked an LLM to write a fix to a bug they encountered, and then submitted it as their own fix, how does one know it is slop?
The Linux kernel gets a ton of PRs constantly. My big issue with LLM code is that it is really good at appearing to be thought out, but actually reverse engineering it can reveal major mistakes.
A good reviewer should still catch them, but it takes longer since the “code smell” is different. Humans will usually make consistent simple mistakes while LLMs will sometimes use advanced patterns to make the same mistake.
The fact that these take longer to review, are usually much longer in general, and are appearing at an incredible pace (1hr to generate, 1 day to review) creates a backlog that becomes unbearable to review.
A lot of open source projects have banned them for this reason, but the ones that do allow them tend to use LLM reviewers to help clear the backlog which is a whole different ouroboros of an issue.
What I hate about the review vs generation disparity is that there’s suddenly pressure on the reviewer (the one gatekeeping poor quality work) to work faster under a greater workload.
If I don’t review it quickly, someone who doesnt give a shit will, and they’ll hit approve on slop and get a gold star for doing it fast. If you make a comment, the contributor suddenly spams 5 commits with loads of changes which need reviewing - all while that same contributor gets to work on (ruin) other shit while you’re stuck with their mess.
In my last job, I could work through a problem with someone to get a good submission, with compromise of course. Since then, I work with someone who swears by AI depsite barely having a grasp of programming and I straight up want to reject half of his PRs. But can I? No, because I’m at the same level as him and get given less respect by management because I’m not churning out slop so I can be free to get invites to other meetings for technical commentary. It’s a pisstake.
I’m very happy that I’m lead/solo dev at my company because I get to just say no. If I don’t want to maintain it it doesn’t get done since everything I do is internal tooling.
Also means that our whole company runs on a Python codebase with a single dependency. It took me 3 years to finally get around to vendoring a fast graphing library in our codebase because I wanted everything to be ours. We’ve had 3 other devs come through and only one of them has actually been able to produce something that got merged in.
Maybe I’m just an insane person, but when all of production relies on our code, I want to know exactly how it works and where everything is.
Ding ding ding. When an AI is spitting out code like a tutorial yet the contributor doesnt know that the comments are unhelpful (or worse, they havent read it at all)
If you ever had the (dis)pleasure of reviewing code from lowest-bidder outsourced developers in the early 00s, you will understand the garbage LLMs try to shovel into your face with the same exact confidence. It’s not just the code that’s sloppy; the commits, communication, and even the original author’s intent[1], can be just as sloppy. This makes review harder because you can’t reliably understand the thread of changes because the LLM is hallucinating[2] the whole thing. I have to not only check if libraries are made up, but if those libraries are hallucinated and typo squatted by hackers. I have to check if they are using other helper functions I’ve already written or if they just make up more! Oh the LLM split up a function into several others that are useless on their own and called it refactoring. Or just spinning and spinning and spinning and spinning…
But the thing most frustrating is how much Roguelike development has become. Is Claude going to fix my problem or TPK my repo? Do I just “need another run” to get to my goal? Better CLAUDE.md? Skills? All this yapping from a RNGesus is turning me into a digital Roman.
that is, if there even is an author under the Simulacrum ↩︎
Yes, LLMs hallucinate everything. Some of those hallucinations are useful. ↩︎
Yes, there is a flavor to LLM code. Generally there is poor structure and comprehensibility, the more code slides right off your brain when you read it despite there being lots of comments the more likely it is to be slop. On the other hand, bad human written code tends to have a feeling of impenetrability that’s often accompanied by a lack of comments.
If the code is good enough you wouldn’t know, but then it’s likely the LLM code was audited and fixed by a skilled human so 🤷♂️.
In my experience there are smells:
Comments are oddly explanatory of basic concepts.
Making an unusually large number of changes in a single commit (specifically, changes that arent related to the goal of the commit)
Writing new infrastructure code that duplicates the work of old code for no reason.
Using old library or language conventions unexpectedly.
Most importantly:
A contributor who has no idea what their code does.
A lot of it you learn from using AI yourself. I use it for inline hints, and i see what goes wrong in that and from testing agentic crap - the stuff that stinks of bad AI use that you can then see in the work of colleagues.
Well the latest models tend to overengineer things and leave a lot of superfluous docs and comments. This can be mitigated if you give the slightest bit of a fuck and actually somewhat read what was generated, I think the best telltale sign is if it’s a new contributor with a large and complex change that would be generally completely out of character for a new unknown contributor.
genuine question from me, the uninformed: If a change or commit is being presented as code a person authored (or without indicating one way or another) how can someone tell it is LLM slop vs code a person wrote?
Like, with writing, there are sort of semi-intangible patterns that make me recognize AI slop, but even then I don’t know for sure- its mostly based on suspicion that comes from reading AI slop in the first place. Like one example I can think of is the “Not X, but Y” summarization at the end of nearly any essay or brief explanation of something. Its always meant to be poetic or impactful and it might have been a genuine style at one point -I think it was common in advertisements maybe?- but now it makes me think, “huh, this is probably slop”
So for code, is there some BS that they are always doing that feels inhuman or wrong? Do they make the same errors or add unnecessary shit that makes people roll their eyes? Or is it just a high volume of work with only a short turnover? If someone asked an LLM to write a fix to a bug they encountered, and then submitted it as their own fix, how does one know it is slop?
The Linux kernel gets a ton of PRs constantly. My big issue with LLM code is that it is really good at appearing to be thought out, but actually reverse engineering it can reveal major mistakes.
A good reviewer should still catch them, but it takes longer since the “code smell” is different. Humans will usually make consistent simple mistakes while LLMs will sometimes use advanced patterns to make the same mistake.
The fact that these take longer to review, are usually much longer in general, and are appearing at an incredible pace (1hr to generate, 1 day to review) creates a backlog that becomes unbearable to review.
A lot of open source projects have banned them for this reason, but the ones that do allow them tend to use LLM reviewers to help clear the backlog which is a whole different ouroboros of an issue.
What I hate about the review vs generation disparity is that there’s suddenly pressure on the reviewer (the one gatekeeping poor quality work) to work faster under a greater workload.
If I don’t review it quickly, someone who doesnt give a shit will, and they’ll hit approve on slop and get a gold star for doing it fast. If you make a comment, the contributor suddenly spams 5 commits with loads of changes which need reviewing - all while that same contributor gets to work on (ruin) other shit while you’re stuck with their mess.
In my last job, I could work through a problem with someone to get a good submission, with compromise of course. Since then, I work with someone who swears by AI depsite barely having a grasp of programming and I straight up want to reject half of his PRs. But can I? No, because I’m at the same level as him and get given less respect by management because I’m not churning out slop so I can be free to get invites to other meetings for technical commentary. It’s a pisstake.
I’m very happy that I’m lead/solo dev at my company because I get to just say no. If I don’t want to maintain it it doesn’t get done since everything I do is internal tooling.
Also means that our whole company runs on a Python codebase with a single dependency. It took me 3 years to finally get around to vendoring a fast graphing library in our codebase because I wanted everything to be ours. We’ve had 3 other devs come through and only one of them has actually been able to produce something that got merged in.
Maybe I’m just an insane person, but when all of production relies on our code, I want to know exactly how it works and where everything is.
my dream is to be this
It occurs eventually once the slop enjoying programmer is de-skilled where they only glance at AI output and start losing track of the bigger picture.
The most obvious one for code is the “comment above a line of code that describes exactly what the code does”
# This does A with B as an input DoA(B)Ding ding ding. When an AI is spitting out code like a tutorial yet the contributor doesnt know that the comments are unhelpful (or worse, they havent read it at all)
afaik LLM code is usually unnecessarily verbose and tries to include libraries that typically wouldn’t be necessary.
But also it seems that after writing work emails, writing code seems to be the thing LLMs do best.
If you ever had the (dis)pleasure of reviewing code from lowest-bidder outsourced developers in the early 00s, you will understand the garbage LLMs try to shovel into your face with the same exact confidence. It’s not just the code that’s sloppy; the commits, communication, and even the original author’s intent[1], can be just as sloppy. This makes review harder because you can’t reliably understand the thread of changes because the LLM is hallucinating[2] the whole thing. I have to not only check if libraries are made up, but if those libraries are hallucinated and typo squatted by hackers. I have to check if they are using other helper functions I’ve already written or if they just make up more! Oh the LLM split up a function into several others that are useless on their own and called it refactoring. Or just spinning and spinning and spinning and spinning…
But the thing most frustrating is how much Roguelike development has become. Is Claude going to fix my problem or TPK my repo? Do I just “need another run” to get to my goal? Better CLAUDE.md? Skills? All this yapping from a RNGesus is turning me into a digital Roman.
that is, if there even is an author under the Simulacrum ↩︎
Yes, LLMs hallucinate everything. Some of those hallucinations are useful. ↩︎
Yes, there is a flavor to LLM code. Generally there is poor structure and comprehensibility, the more code slides right off your brain when you read it despite there being lots of comments the more likely it is to be slop. On the other hand, bad human written code tends to have a feeling of impenetrability that’s often accompanied by a lack of comments.
If the code is good enough you wouldn’t know, but then it’s likely the LLM code was audited and fixed by a skilled human so 🤷♂️.
In my experience there are smells:
Most importantly:
A lot of it you learn from using AI yourself. I use it for inline hints, and i see what goes wrong in that and from testing agentic crap - the stuff that stinks of bad AI use that you can then see in the work of colleagues.
Well the latest models tend to overengineer things and leave a lot of superfluous docs and comments. This can be mitigated if you give the slightest bit of a fuck and actually somewhat read what was generated, I think the best telltale sign is if it’s a new contributor with a large and complex change that would be generally completely out of character for a new unknown contributor.