You could create a way to measure the idea of capability that would bear that out but from a pure discrete mathematics perspective, no, you only get better with a larger memory footprint.
There’s a lot of ways to make that faster or make that behave like a process running on a bigger memory footprint, but ultimately that’s the constraint.
And companies competing in the field of ai can’t justify the expense of cutting down their gigantic model to only know how to identify wood because that has a known and limited impact. They already said they’re shooting for unlimited immeasurable impact on the scale of replacing all human labor and got massive funding for it.
It doesn’t matter if it’s easier to do one backflip, you asked me to triple dog dare you to do a million backflips. Well… we’re waiting!
Again, there is no reason to think that you can just keep making the model bigger and keep getting improved capability that way. In fact, we already know that’s not the case because simply making them bigger stopped being the focus. The real breakthrough is going to come from better algorithms.
You said there’s no reason to think you can just keep making the model bigger and keep getting improved capability.
there’s the structure of the neural network itself. Fundamentally, adding nodes and layers increases the ability of the model to handle more complex input.
Then there’s the actual models we see in use. They are literally as large as the hardware allows. The only reason to use smaller models are to fit some constraint.
So both by the book and in practice bigger is always better.
Now we can’t always go big. I can’t afford to purchase a dgx or even upgrade my wiring to power it, let alone pay the power bill it would rack up or all the other utilities alone when my wife leaves me because of the sound.
My computer can only fit so many expansion cards and pcie is so slow compared to hbm that I’m better off running a small model quickly that fits on one card as opposed to a larger one slowly across several cards.
But those are all constraints. When I replace my motherboard with supermicro gpu host fabric I no longer am limited by the pcie bandwidth and can quickly use models that fit across several cards.
I do agree with you that the future is smaller models, not because of the fundamental nature of the concepts involved but because of the complex constraints that are coming into play.
The problem is with the context and data propagation through the network. As you keep making it bigger it becomes slower and less focused. And there is research showing that smaller models do outperform large ones on some tasks https://cacm.acm.org/news/bigger-not-necessarily-better
What I expect we’ll see going forward is more hierarchical architecture where you have finely tuned models for specific tasks with a general routing model on top. This is basically already where MoE architecture is moving now. We might also see stuff like neurosymbolics get more popular where the LLM acts as a stochastic engine within a symbolic logic system. The model can handle noisy input from the real world, and transform it into structured data that a symbolic engine can operate on.
Brute forcing the problem is a naive approach and US labs took it because they effectively had unlimited resources to train their models until now.
And when more compute becomes available, solutions that are more efficient are going to further benefit from that as well. We see this with DeepSeek right now. They focused on efficiency over capability up front, and now they have a fundamentally cheaper architecture that’s rapidly catching up in capability.
And to bring it aaaaaalllll the way back to what your post is about: the industry signaling it would accept a limitation of some sort means that it predicts (using privileged information available to it via deep connections at all levels from design to manufacturing marketing and sales!) it will be unable to meet current demand with new hardware over the time that new hardware will be viable.
They’d have to raise prices.
And the only reason they wouldn’t want to raise prices is because they wouldn’t be profitable if they did.
Openai cancelled ipo, trumpbucks 2.0, etc. are foreshocks.
I think it’s a little disingenuous to imply that deepseek is a small model. Even flash requires 160gb of vram and it’s important to note that’s a size aimed at a constraint, the vram density of hbm equipment from two generations ago. (And the density allowed by using last generations consumer equipment plugged into ten amp circuits with doubled vram see all those 48gb 4090s floating around!)
They didn’t arbitrarily decide that they’d use a smaller model, they were targeting a constraint. Which of course was really juicy because of all the demand for text inference and the limited new hardware to fulfill it.
The paper you cite is true, I haven’t read it but it based on the title it can’t really be wrong unless they just get wildly over their skis with their claims…
You’re right that everything’s gotta walk through the layers and more layers means a longer walk. That’s limited by memory and interconnect bandwidth though which is still doubling or close to it every year or close to it.
Which means it would have to be twice as fast to use a smaller model on cutting edge hardware to be a real “wall”.
We are not near the end of the memory bandwidth road yet, quantum tunneling isn’t rearing its head again as the workaday wafers that ferry serialized streams from place to place get faster and more numerous. Because they don’t need to get smaller really. The port of New York can grow and sprawl and sprout heretofore unseen support structures like coolers and voltage regulators and coolers for its voltage regulators.
I’ve seen in action what you describe though. On a, sensible chuckle, tiny card like a 3050 slotted into a system with pcie3 very small models run faster than those who can barely fit in the vram with their little bitty context, but that difference shrinks significantly when the card is a 5060 or something that still has limited vram but can load faster due to a faster pcie interconnect.
Now that may seem like apples and oranges because it’s two completely different things but the point of the comparison is to show that the difference between performance measures in time to first, tps or whatever other measurement might be in vogue at the moment when two models are compared shrinks when the interconnect and memory bandwidth gets faster.
To butcher a car metaphor, you can turbocharge an ls and get more power but you’re not escaping engine wear=crankshaft rotations. That might not be as butchered as it first seemed even though the domains are all shifted.
I don’t disagree with any of that. But I think we’re talking about different things here. My point is that it’s not clear that capability will continue to scale in a useful way just because you make the model bigger. If you keep getting diminishing returns while needing vastly more resources, then it’s not economically viable to run these huge models.
So, I expect that labs focusing on more efficient architectures will outcompete those that are trying to brute force the problem. Like sure, DeepSeek isn’t small in a sense that you can run it locally, but it is small compared to other models in its class, and much more energy efficient. Whatever hardware we get down the road is going to benefit more efficient models the same way meaning that they will always have a competitive advantage.
From what I see in the latest releases from Anthropic, Fable isn’t a huge leap ahead from Opus. There is an improvement, but it’s not a definitive jump in capability the way it was from Sonnet to Opus. So, they managed to make a bigger model, but got diminishing returns, and it’s evidently so expensive to run right now that they can’t even offer it as a default.
The real progress will almost certainly be happening in hybrid architectures where people start coming up with algorithms that complement LLMs and augment their capabilities. These will be like different brain regions responsible for different tasks. For example, memory formation is an obvious example here, another would be to have a built in mathematics engine. A real huge win would be to figure out how to do few shot learning on the fly as well, for which memory is a prerequisite. So, there are plenty of things we already know that can be done much better.
You could create a way to measure the idea of capability that would bear that out but from a pure discrete mathematics perspective, no, you only get better with a larger memory footprint.
There’s a lot of ways to make that faster or make that behave like a process running on a bigger memory footprint, but ultimately that’s the constraint.
And companies competing in the field of ai can’t justify the expense of cutting down their gigantic model to only know how to identify wood because that has a known and limited impact. They already said they’re shooting for unlimited immeasurable impact on the scale of replacing all human labor and got massive funding for it.
It doesn’t matter if it’s easier to do one backflip, you asked me to triple dog dare you to do a million backflips. Well… we’re waiting!
Again, there is no reason to think that you can just keep making the model bigger and keep getting improved capability that way. In fact, we already know that’s not the case because simply making them bigger stopped being the focus. The real breakthrough is going to come from better algorithms.
You said there’s no reason to think you can just keep making the model bigger and keep getting improved capability.
there’s the structure of the neural network itself. Fundamentally, adding nodes and layers increases the ability of the model to handle more complex input.
Then there’s the actual models we see in use. They are literally as large as the hardware allows. The only reason to use smaller models are to fit some constraint.
So both by the book and in practice bigger is always better.
Now we can’t always go big. I can’t afford to purchase a dgx or even upgrade my wiring to power it, let alone pay the power bill it would rack up or all the other utilities alone when my wife leaves me because of the sound.
My computer can only fit so many expansion cards and pcie is so slow compared to hbm that I’m better off running a small model quickly that fits on one card as opposed to a larger one slowly across several cards.
But those are all constraints. When I replace my motherboard with supermicro gpu host fabric I no longer am limited by the pcie bandwidth and can quickly use models that fit across several cards.
I do agree with you that the future is smaller models, not because of the fundamental nature of the concepts involved but because of the complex constraints that are coming into play.
The problem is with the context and data propagation through the network. As you keep making it bigger it becomes slower and less focused. And there is research showing that smaller models do outperform large ones on some tasks https://cacm.acm.org/news/bigger-not-necessarily-better
What I expect we’ll see going forward is more hierarchical architecture where you have finely tuned models for specific tasks with a general routing model on top. This is basically already where MoE architecture is moving now. We might also see stuff like neurosymbolics get more popular where the LLM acts as a stochastic engine within a symbolic logic system. The model can handle noisy input from the real world, and transform it into structured data that a symbolic engine can operate on.
Brute forcing the problem is a naive approach and US labs took it because they effectively had unlimited resources to train their models until now.
And when more compute becomes available, solutions that are more efficient are going to further benefit from that as well. We see this with DeepSeek right now. They focused on efficiency over capability up front, and now they have a fundamentally cheaper architecture that’s rapidly catching up in capability.
And to bring it aaaaaalllll the way back to what your post is about: the industry signaling it would accept a limitation of some sort means that it predicts (using privileged information available to it via deep connections at all levels from design to manufacturing marketing and sales!) it will be unable to meet current demand with new hardware over the time that new hardware will be viable.
They’d have to raise prices.
And the only reason they wouldn’t want to raise prices is because they wouldn’t be profitable if they did.
Openai cancelled ipo, trumpbucks 2.0, etc. are foreshocks.
exactly right
I think it’s a little disingenuous to imply that deepseek is a small model. Even flash requires 160gb of vram and it’s important to note that’s a size aimed at a constraint, the vram density of hbm equipment from two generations ago. (And the density allowed by using last generations consumer equipment plugged into ten amp circuits with doubled vram see all those 48gb 4090s floating around!)
They didn’t arbitrarily decide that they’d use a smaller model, they were targeting a constraint. Which of course was really juicy because of all the demand for text inference and the limited new hardware to fulfill it.
The paper you cite is true, I haven’t read it but it based on the title it can’t really be wrong unless they just get wildly over their skis with their claims…
You’re right that everything’s gotta walk through the layers and more layers means a longer walk. That’s limited by memory and interconnect bandwidth though which is still doubling or close to it every year or close to it.
Which means it would have to be twice as fast to use a smaller model on cutting edge hardware to be a real “wall”.
We are not near the end of the memory bandwidth road yet, quantum tunneling isn’t rearing its head again as the workaday wafers that ferry serialized streams from place to place get faster and more numerous. Because they don’t need to get smaller really. The port of New York can grow and sprawl and sprout heretofore unseen support structures like coolers and voltage regulators and coolers for its voltage regulators.
I’ve seen in action what you describe though. On a, sensible chuckle, tiny card like a 3050 slotted into a system with pcie3 very small models run faster than those who can barely fit in the vram with their little bitty context, but that difference shrinks significantly when the card is a 5060 or something that still has limited vram but can load faster due to a faster pcie interconnect.
Now that may seem like apples and oranges because it’s two completely different things but the point of the comparison is to show that the difference between performance measures in time to first, tps or whatever other measurement might be in vogue at the moment when two models are compared shrinks when the interconnect and memory bandwidth gets faster.
To butcher a car metaphor, you can turbocharge an ls and get more power but you’re not escaping engine wear=crankshaft rotations. That might not be as butchered as it first seemed even though the domains are all shifted.
I don’t disagree with any of that. But I think we’re talking about different things here. My point is that it’s not clear that capability will continue to scale in a useful way just because you make the model bigger. If you keep getting diminishing returns while needing vastly more resources, then it’s not economically viable to run these huge models.
So, I expect that labs focusing on more efficient architectures will outcompete those that are trying to brute force the problem. Like sure, DeepSeek isn’t small in a sense that you can run it locally, but it is small compared to other models in its class, and much more energy efficient. Whatever hardware we get down the road is going to benefit more efficient models the same way meaning that they will always have a competitive advantage.
From what I see in the latest releases from Anthropic, Fable isn’t a huge leap ahead from Opus. There is an improvement, but it’s not a definitive jump in capability the way it was from Sonnet to Opus. So, they managed to make a bigger model, but got diminishing returns, and it’s evidently so expensive to run right now that they can’t even offer it as a default.
The real progress will almost certainly be happening in hybrid architectures where people start coming up with algorithms that complement LLMs and augment their capabilities. These will be like different brain regions responsible for different tasks. For example, memory formation is an obvious example here, another would be to have a built in mathematics engine. A real huge win would be to figure out how to do few shot learning on the fly as well, for which memory is a prerequisite. So, there are plenty of things we already know that can be done much better.