putting aside the contents of the graph, trying to read the name of each model is absolutely horrendous
A lot of people have deep misunderstandings on what the massive data centers are for. While they do use a decent portion of them to run requests, as they’re trying to service a lot of people, a big chunk is training and data processing. Not for the end user, but for their R&D, model production, and the endless churn of data processing as they find and create new data to incorporate in models.
Which is how you can run pre-trained LLMs on your computer, if it’s a good one.
I mean, China is training comparable models with a tiny fraction of compute. And it’s not just them distilling US models.
that still doesn’t mean inference is profitable. (which isn’t to say i give a shit about profits, but it’s a good indicator of economic viability even if socialized). inference still requires far more compute to complete a task than most previous means of doing so.
Claude has always been better. But I am so happy for open source llms
I think there’s an upper limit of how good you need a model to be for most tasks, and if local models can do another jump like this where they’re as good as current Claude is, then there’s really not much reason to rely on services at that point.
Something I think would help accelerate this is educating people on how to run them. But maybe that comes after a long period of stability, since models leapfrog each other every few months now.
Like, first, which model do I run? Depending on your hardware, you’ll get 50 answers and random links to huggingface. Are the things they are linking models from companies or organizations you hear about? Maybe, or maybe they’re some strange distillation or uncensored/abliterated model, or some custom tweak that nobody can explain without using a bunch of nonsensical sounding terms.
Second, what are they actually capable of? What other apps can they link to and how do you link them? The second part is not always easy. A friend recently started using them to assist with organizing his life and thoughts by connecting it to Obsidian and Zotero. He has ADHD and says it helps him immensely in actually getting those ideas down, if not developing them further. I never would have considered this kind of workflow.
Third, what software do I use? LM Studio? Ollama? Ramalama? How come some connect to my IDE, but not others? What is rocm and why do some support it and some not? Do I need that?
These are rhetorical questions, but it’s honestly a swamp trying to get clear answers to them online and probably the biggest barrier to getting people to use these local models at the moment.
usually the merge models, uncensored ones, abliterated, etc are complete dogshit. if you can, run unsloth ggufs on llama.cpp and anything of vital importance in transformers
For sure, I think having an easy to follow process for people to get set up would be really useful. I can see how all of this can be pretty overwhelming, and unless you know what you can do with these things, it might not even feel like it’s worth the bother.
I expect in a year or so things will likely start to get a bit more stable, and the churn will slow down enough for some standard solutions to start emerging. Even if the models keep changing, having a standardized harness to drop them into would go a long way towards making this tech accessible.
Given that we’re nearly at the point where decade-old GPUs can run close to the best models, I think consumer-grade “LLM machines” are entirely plausible if hardware prices can ever stabilize. Could just be an eGPU and snazzy software stack if someone already has a laptop.
With how much “AI” companies want to charge an eGPU would already pay for itself in a few days for just one person at a ridiculous “AI-first” company.
Yeah now that hardware isn’t as much of an issue (I got a couple models working on my years-old android) my barrier is pretty much as you’ve described. I can’t think of a personal usecase since I don’t code much, and as such delving into all the complexities is a big ask as of right now.
So far the only utility I regularly get out of LLMs is searching for internet results, but I have no idea how to go about getting a local model to do so, or if it’s even possible with old hardware.
Spent like an hour messing around on an old Linux desktop with an rx580, but never figured it out lol
im pretty excited for qwen 3.7, the 35b models always blow me away with each new release
Stinky shit from last year now stinks less then the new shit we have been producing! Now with more corn too!
I run a 9B model on an $80 GPU from 10 years ago, and that’s all I need
You can run it on your desktop if you have a shit ton of vram. There are more compact versions, but the full model is 55.6 GB. You could run it on an Nvidia H100 80GB, which runs for $40k. Or more likely you’d run it on a Mac Studio M3 Ultra with 96GB of unified memory, which still runs for $4k.
im running it on my desktop with about 30 t/s on a computer from 2 years ago with a tpu bolted on. im really good at researching things, its my job. i do science™️. its a little worse at researching things online than i am and rarely hallucinates. its especially good with unlimited search toolcalls. i was able to debug a problem in my code that would’ve taken me all day to figure out but after quite a lot of wiresharking and pushing logs to qwen i was able to resolve the problem in 45 minutes. its also often able to find unpaywalled scientific articles that are hidden in every major search engine which is very useful when anna’s archive can’t pull through.
these things are getting fairly impressive to me and i am generally very skeptical of them and skim everything they do. qwen is doing some tremendous work especially in democratizing them as general use tools. 30 t/s isn’t great, I have to wait quite a bit for it to think, but its better than feeding a ton of my data to some lab somewhere.
This is really neat to hear. Can I ask which version of the model you’re running? And which TPU you’re using? If it’s usable for work I’d be happy to poke my bosses about letting me use a local model.
i’ve been using qwen 3.6 35b a3b, 64gb ram, a used 4080, llama.cpp with some tweaks for more tool calling, and some amd cpu that has a npu in it from a couple years back i cant be assed to boot it up right now to check lol
i got the ram and gpu before everything went completely to shit, whole setup was like 2k usd or something but was kinda important to my job so i got it with a partial rebate from my employer. i havent really tried to optimize it much. qwen 35b a3b has slightly worse performance than 27b but runs a lot faster.
Running fp16, in current year?
https://huggingface.co/unsloth/Qwen3.6-27B-GGUF
even 5ks is 19gb. Plus context but that still fully fits on a 3090. But even splitting it across gpu + cpu isn’t as bad as it could be.
The cheapest way to run larger models locally is the Vega architecture Radeon Instinct MI60. 32gb of HBM for 3-500 usd used, add 30-40 per unit for active cooling.
Idk, I’m not sure whether having this ubiquity of associative content generation is a good thing.
What outcome are you looking for… taking away the human input from a human interaction like an email? A workaround for skimming and reading comprehension? Something that helps us break even in sifting through the slop that there’s now hundreds of times more of in quantity? Allowing people to write code without understanding it or thinking about it?
These tools are incredibly useful for programming tasks and automating tedious work. This whole idea of gatekeeping is just so offensive to be honest. Believe it or not, but not everyone has the time to spend years to learn coding. If a tool can solve a problem for somebody that’s a genuinely good thing. This whole narrative of how you should just do everything manually is so tiring. Why do you even bother using a computer at all if you’re so against automation?
Why do you even bother using a computer at all if you’re so against automation?
That’s a bad faith argument and you know it, I was just talking about writing an email (instead of handwriting and mailing a letter) and specified the ubiquity of automation, not its existence.
I would be mostly* fine using a level of technology where “AI” meant scripted logic rather than data mining everything for a most likely answer… just like it did before 2022. Mostly, because we’re on a crash course with the limits to growth that cannot be solved automatically by having a communist party at the helm.
What I find tiring is the incessant hype around a kind of agency that paradoxically seems to be slipping further and further away from us. That is the last 10 years in a nutshell, especially in tech.
Previous thoughts on how I do not want this AI.
You were talking about avoiding using LLMs to write code. But more fundamentally, it seems like you’ve just drawn an arbitrary line here at a level of automation you’re familiar with. And any further automation is somehow bad. And that was precisely my point. How did you decide that your current use of computers is an appropriate level of automation? Why use a spell checker, or autocomplete, or even us a computer to write when you could go back to pen and paper?
I also have no idea how we’re at the limits of growth here, or what this statement is even based on. You’re just stating random assertions as facts here.
The rant you linked really just comes down to the world is different from what I’m used to and I don’t like it.
Again with the strawman. Having reasons to dislike one technology is not tantamount to hating all technology. Autocomplete is fine as assistive technology for people who can’t type faster than 15 WPM. Likewise, a spellchecker is a project that has a very clear end, a very clear benefit, it can easily be transparent in how it works, and does not require you to put everything through a brute-force recursive association. There are aspects to technologies that we can appraise and discern whether they’re beneficial to us or not.
I don’t celebrate cars either, even if they are electric and Chinese. They flatten and alienate the physical environment in a remarkably analogous way to how LLMs flatten and alienate the digital environment.
Vibe code has a lot of externalities, it is slower and less resource efficient in ways that accumulate as further vibe coding goes on, and it doesn’t build or hone people’s skills. Use of LLMs has directly been shown to diminish attentional and critical thinking capacity. Hell, I remember you personally arguing not too long ago that “something that systematically disincludes the human input is a fascist aesthetic”.
We have brought on the age of the mass deepfakes, while cultivating a process that materially/logistically cannot be sustained by the Earth’s resources for another 100 years, by looking at tech without broad long-term considerations. Diving faster headfirst into it will not solve our problems.
Data centers (which you can’t have LLMs without) were largely bad for humanity before 2022, but in a way that could have possibly been amended by socialist governance. At this point they’re just bad for everything except short-term market speculation. The world has not gotten any better in the past 5 years despite spending more on compute and on AI research by several orders of magnitude. It’s not going to fix our problems, it’s just going to be a crutch that also creates new ones.
If you keep arguing that personal LLMs/genAI are a useful and adaptive thing for society, I might just as well outsource to one and have you talk to it instead. What a great forum we would have then.
There is no straw man, I pointed out that you draw an arbitrary line at technology you’re familiar with. Nothing you’ve said here contradicts my point.
Also, you love making these sweeping statements like vibe code is slower and less resource efficient. Evidently you haven’t actually seen most code that humans write, because boy would you be shocked then. What LLMs do however is allow people who have no coding skills to solve problems and automate things in a way they couldn’t before unless they paid thousands of dollars to a developer. Meanwhile, these studies you reference use laughably small samples, but since they fit your narrative you choose to cite them uncritically as facts. And I’m not even going to touch your Reductio ad Hitlerum there.
The whole energy use argument is likewise hilarious given the context of this technology having become dramatically more efficient in just a single year. Not to mention that we were already building data centers for shit like bitcoing mining for decades. The actual problem here is capitalism but you’re just fixating on how a particular technology is being abused under it instead of doing any sort of systemic analysis.
Honestly, if you used one, it might’ve come up with a more coherent argument.
How is life qualitatively or quantitatively better now that we have access to these things? What were we all going without, for our entire lives, that is a noticeable net improvement, and that isn’t swallowed up by increased demand and noise?
It’s one thing to make up rubrics and say that the models are scoring ever higher on them, while “the graph goes up and to the right”. But on a practical level, what was so wrong or unbearable with the world before that introducing an LLM would revolutionize for the better? What software was so broken, or what process was so inaccessible, that the raw computational approach became the best solution? I’m actually open to hearing about this.
There are tons of ways LLMs have already improved daily life. Before LLMs, trying to translate a document required either hiring a translator or wrestling with clunky rule-based systems that often produced gibberish. Today, a model delivers usable translations in seconds. These might not be perfect, but they’re far ahead of anything that was possible before. And these things are also helping people with disabilities, somebody with dyslexia or a blind professional had to wait days for accessible documents. Now, screen readers paired with LLMs can summarize, rephrase, and read aloud any PDF in real time. Researchers spent countless hours skimming through hundreds of pages of regulatory filings or legal opinions to find a single relevant clause. Now, they ask a question and have a machine find the answer with citations. These are just a few examples of genuine and cumulative improvements that free people from drudgery and lower barriers for those who were previously locked out.
And here’s a concrete illustration of how this tech can be used on the left to level the playing field allowing regular people to do what used to take a whole team of analysts before. https://www.youtube.com/watch?v=KQog2g_zoJ8
Examples of this tech being useful are only hard to find for people who choose to be aggressively ignorant on the subject.
