I dont think a local model will do 99% of what the big ones can, and even if it could, it would require an enormous amount of memory and cpu to do it.
People pay for a good user interface and convenience, that’s well known. Netflix and Spotify are built on that idea and people starting using them even over free pirated stuff :)
I’ve been consuming lots of “build your own local AI coding agent” kind of content lately. No doubt this is the same as the guys that are welding together their own versions of Shelby Cobras and stuff… but in a few years you’ll be able to buy a “server” with enough GPU and ram to host strong open models and serve them to your little office.
I work in a small company, and if AI plays out the way Anthropic sees it’d their AI bill will be $100k every year (fire 2-3 people, pay for AI so that the remaining people do more). At that level of expenditure, why wouldn’t my poss pay as much as a copy machine $25k, lets say, go give “local AI” a chance?
The stock valuations that most tech companies carry (and certainly the 2-3 trillion that is being talked about for Anthropic) speak to potential future revenue.
To me, I see computing and actual “model” advances meeting the need within a few years… if not by entrenched companies like Nvidia, then by insurgent chipmakers or new (Chinese or otherwise) chipmakers.
In 10 years you’ll have “opus” is not better running in your phone and probably something more powerful in a PC.
Qwen3.8-flash-next broke the vram requirement with its new architecture (which is a preview of their 4.0 architecture). You can run a 177b parameter model on 12gb VRAM, 64gb ram and 80gb SSD at 30tok/s and the speed scales almost linearly with increases to vram.
Incidentally, this was proven at almost the exact time all the calls for “regulating local models because they’ll spread into the toasters” started.
You absolutely can have local models that run on consumer grade hardware, and the proprietary companies know it and are scared to death.
Are these really able to be as good as chat gpt? Because every time I’ve tried a model like that, it has not been able to be nearly as good. I have 48 gb memory on my machine so I can’t run this particular one though.
I would buy more ram and a whole new computer if the market wasn’t insane.
I won’t pretend to be an expert but my understanding is that the architecture loads experts from the SSD on demand with the RAM serving as a hot cache and the VRAM running the active experts.
As I said above I was able to run 70tok/s on my 4090w64GB ram and a samsung pcie4.0 ssd.
I haven’t run the benchmarks myself, but I can say the quality of response I was getting seemed on par with the frontier models I use for work (Claude opus/fable) and the published benchmarks for the model compete with those same models.
It is a q3 quantization so there is some small loss of accuracy, but from what I’ve understood those are modest losses in the worst case.
There are models that don’t need to consume the entire worlds data infinitively. We’ve had then before llms, things like the pi ai camera can do ai processing on image with something like 8mb of on board storage.
The problem is anthropic, open ai, who ever are trying to do the stupidist approach and assume they can brute force money and power at the problem and don’t understand that they need exponentially infinite data in a way that is unobtainable.
In an ideal world there would be a million little small local models that so specific tasks and can be ran on consumer hardware.
I dont know, maybe, but I think either way, local models will be a fringe interest for technical people, and not something that will affect the income for these big tech Ai corps in any significant way.
Because you simply can’t run a model with all the worlds knowledge on your home computer.
Someone said that maybe there will be small specialized models that only know about a certain thing, and sure, that’s possible.
You also can’t run a model with all the worlds knowledge on a supercomputer. That simply doesn’t exist. To think that even the largest frontier models can do that is a mistake.
The real question is: will the median user be able to run basically everything they need locally, and how much compute would they need to do that? These companies are betting the farm that it will never happen, and that they’ll be able to charge whatever they want in rent to a captive audience.
But I think these tech CEOs and ML researchers talking about “AI killing us all” is related via the aphorism: It’s easier to imagine the end of the world than the end of capitalism. They can see the writing on the wall, but can’t imagine a future for themselves.
I dont think a local model will do 99% of what the big ones can, and even if it could, it would require an enormous amount of memory and cpu to do it.
People pay for a good user interface and convenience, that’s well known. Netflix and Spotify are built on that idea and people starting using them even over free pirated stuff :)
Maybe not “today” but today is still early days.
I’ve been consuming lots of “build your own local AI coding agent” kind of content lately. No doubt this is the same as the guys that are welding together their own versions of Shelby Cobras and stuff… but in a few years you’ll be able to buy a “server” with enough GPU and ram to host strong open models and serve them to your little office.
I work in a small company, and if AI plays out the way Anthropic sees it’d their AI bill will be $100k every year (fire 2-3 people, pay for AI so that the remaining people do more). At that level of expenditure, why wouldn’t my poss pay as much as a copy machine $25k, lets say, go give “local AI” a chance?
The stock valuations that most tech companies carry (and certainly the 2-3 trillion that is being talked about for Anthropic) speak to potential future revenue.
To me, I see computing and actual “model” advances meeting the need within a few years… if not by entrenched companies like Nvidia, then by insurgent chipmakers or new (Chinese or otherwise) chipmakers.
In 10 years you’ll have “opus” is not better running in your phone and probably something more powerful in a PC.
Qwen3.8-flash-next broke the vram requirement with its new architecture (which is a preview of their 4.0 architecture). You can run a 177b parameter model on 12gb VRAM, 64gb ram and 80gb SSD at 30tok/s and the speed scales almost linearly with increases to vram.
Incidentally, this was proven at almost the exact time all the calls for “regulating local models because they’ll spread into the toasters” started.
You absolutely can have local models that run on consumer grade hardware, and the proprietary companies know it and are scared to death.
Are these really able to be as good as chat gpt? Because every time I’ve tried a model like that, it has not been able to be nearly as good. I have 48 gb memory on my machine so I can’t run this particular one though.
I would buy more ram and a whole new computer if the market wasn’t insane.
Interesting.
i would not recommend using the SSD to run local LLMS,its slow
best: VRAM only
good: VRAM + RAM
bad: Swapping
I don’t think they’re insinuating running inferences from the SSD. But you need somewhere to store the weights for the GPU to load.
i am trying to say: you need the SSD to store it but not swap to it due to its slowness
I won’t pretend to be an expert but my understanding is that the architecture loads experts from the SSD on demand with the RAM serving as a hot cache and the VRAM running the active experts.
As I said above I was able to run 70tok/s on my 4090w64GB ram and a samsung pcie4.0 ssd.
I haven’t run the benchmarks myself, but I can say the quality of response I was getting seemed on par with the frontier models I use for work (Claude opus/fable) and the published benchmarks for the model compete with those same models.
It is a q3 quantization so there is some small loss of accuracy, but from what I’ve understood those are modest losses in the worst case.
There are models that don’t need to consume the entire worlds data infinitively. We’ve had then before llms, things like the pi ai camera can do ai processing on image with something like 8mb of on board storage.
The problem is anthropic, open ai, who ever are trying to do the stupidist approach and assume they can brute force money and power at the problem and don’t understand that they need exponentially infinite data in a way that is unobtainable.
In an ideal world there would be a million little small local models that so specific tasks and can be ran on consumer hardware.
maybe they can do like 60% of the big ones?
I dont know, maybe, but I think either way, local models will be a fringe interest for technical people, and not something that will affect the income for these big tech Ai corps in any significant way.
Because you simply can’t run a model with all the worlds knowledge on your home computer.
Someone said that maybe there will be small specialized models that only know about a certain thing, and sure, that’s possible.
You also can’t run a model with all the worlds knowledge on a supercomputer. That simply doesn’t exist. To think that even the largest frontier models can do that is a mistake.
The real question is: will the median user be able to run basically everything they need locally, and how much compute would they need to do that? These companies are betting the farm that it will never happen, and that they’ll be able to charge whatever they want in rent to a captive audience.
But I think these tech CEOs and ML researchers talking about “AI killing us all” is related via the aphorism: It’s easier to imagine the end of the world than the end of capitalism. They can see the writing on the wall, but can’t imagine a future for themselves.