Me: I need some new hardware to run my transformers on
Mom: We have transformers at home
The transformers at home:
Joking aside, I think you’re right, using discrete floating point math to simulate a transformer architecture will never be able to approach the efficiency of a “native” analog system like actual brains. I think eventually we’ll see someone come up with a hardware transformer that doesn’t require full synchronized clock signals.
Jokes aside, I think a brain and a computer solve two different problem sets. A computer needs to be exact, deterministic. A brain needs to be practical, very much at the expense of precision.
I wouldn’t want a brain to do my math calculations for me… brains suck at those, that’s why I use a computer.
If it didnt introduce a spectrum of new issues, I’d wager we could start encoding data into more complex structures than binary. Like using continuous values instead of just on/off. But, from what I understand, that makes the problem of computing damn near impossible and unaffordable.
Maybe we can find ways to encode more data into the same volume of transistors. Like, if current state could imply additional information. But that would probably impede on performance, as I’m sure the data structures used in a CPU prioritize performance not “adding more implied data.”
Oh, I guess I read transistor. Not transformer. Thanks!
Yeah, a brain does exactly what it’s supposed to do. I can’t imagine how simulating something with alternative hardware could be any more efficient than the original, especially when that new hardware wasn’t designed to do the same thing.
That’s still talking about transistors, though. Even more directly, transformers are just translating text into high dimensional arrays where the semantic structure is captured (relative to all other possible embedding values). It’s an interesting approach to navigating semantic information, but there’s not any guarantee that our brain does the same thing. Either way, I’d bet our brain is doing its job without much accidental complexity, whereas modern transformers will always have the added complexity of encoding / decoding semantic information using hardware not designed for it.
Computer should maybe try staying in its lane. Leave the cognitive dissonance and delusional confidence to the experts.
Yeah, a brain does exactly what it’s supposed to do. I can’t imagine how simulating something with alternative hardware could be any more efficient than the original, especially when that new hardware wasn’t designed to do the same thing.
That’s what Neuralink is for! I can see it now:
The jobs of the future will involve 12 hour shifts serving as a biological GPU. A large helmet is strapped to your head. It remotely, non-invasively takes over your neurons and forces them to perform calculations commanded by a remote server, rather than your own thoughts. It doesn’t override the vital parts of the brain involved in say, keeping your heart beating. But it does override most everything else. And it’s not like a dream. You’re not experiencing a virtual reality in there. For 12 hours, all of your senses are just full static at maximum volume. The experience would quickly drive you mad, if your own conscious awareness was not also dissolved into so much computational grist. You’re essentially a vegetable for 12 hours. Your shift is performed wearing an adult diaper. This is nearly the only form of wage labor available. Not even jobs like janitorial are available. One human mind, suitably harnessed, can remotely pilot a dozen humanoid androids. Half of your coworkers are there involuntarily, unpaid, caught up in one of anti-vagrancy law or another.
Yeah, I said transformer because that seems to be the state of the art in AI architectures, but purpose built neural network hardware might not actually benefit from the same architecture.
A neural network made from analog hardware could theoretically replace a significant portion of an LLM’s processing and not be limited by things like floating point precision or clock speeds. Who needs floating point when you can literally just multiply voltages together with a couple transistor junctions at the speed of light?
There is no way any breakthrough will be made with new technologies whilst the current AI speculation continues over RAM and GPUs, there seems to be still way to much money to be made in this bubble so most of capital resources in this area are being directed tô those “AI” initiatives.
Me: I need some new hardware to run my transformers on
Mom: We have transformers at home
The transformers at home:
Joking aside, I think you’re right, using discrete floating point math to simulate a transformer architecture will never be able to approach the efficiency of a “native” analog system like actual brains. I think eventually we’ll see someone come up with a hardware transformer that doesn’t require full synchronized clock signals.
Jokes aside, I think a brain and a computer solve two different problem sets. A computer needs to be exact, deterministic. A brain needs to be practical, very much at the expense of precision.
I wouldn’t want a brain to do my math calculations for me… brains suck at those, that’s why I use a computer.
If it didnt introduce a spectrum of new issues, I’d wager we could start encoding data into more complex structures than binary. Like using continuous values instead of just on/off. But, from what I understand, that makes the problem of computing damn near impossible and unaffordable.
Maybe we can find ways to encode more data into the same volume of transistors. Like, if current state could imply additional information. But that would probably impede on performance, as I’m sure the data structures used in a CPU prioritize performance not “adding more implied data.”
Hmm… tough one.
They’re talking about AI datacenters though… The whole point is they’re not exact and deterministic, but usually about right.
Oh, I guess I read transistor. Not transformer. Thanks!
Yeah, a brain does exactly what it’s supposed to do. I can’t imagine how simulating something with alternative hardware could be any more efficient than the original, especially when that new hardware wasn’t designed to do the same thing.
That’s still talking about transistors, though. Even more directly, transformers are just translating text into high dimensional arrays where the semantic structure is captured (relative to all other possible embedding values). It’s an interesting approach to navigating semantic information, but there’s not any guarantee that our brain does the same thing. Either way, I’d bet our brain is doing its job without much accidental complexity, whereas modern transformers will always have the added complexity of encoding / decoding semantic information using hardware not designed for it.
Computer should maybe try staying in its lane. Leave the cognitive dissonance and delusional confidence to the experts.
That’s what Neuralink is for! I can see it now:
The jobs of the future will involve 12 hour shifts serving as a biological GPU. A large helmet is strapped to your head. It remotely, non-invasively takes over your neurons and forces them to perform calculations commanded by a remote server, rather than your own thoughts. It doesn’t override the vital parts of the brain involved in say, keeping your heart beating. But it does override most everything else. And it’s not like a dream. You’re not experiencing a virtual reality in there. For 12 hours, all of your senses are just full static at maximum volume. The experience would quickly drive you mad, if your own conscious awareness was not also dissolved into so much computational grist. You’re essentially a vegetable for 12 hours. Your shift is performed wearing an adult diaper. This is nearly the only form of wage labor available. Not even jobs like janitorial are available. One human mind, suitably harnessed, can remotely pilot a dozen humanoid androids. Half of your coworkers are there involuntarily, unpaid, caught up in one of anti-vagrancy law or another.
Yeah, I said transformer because that seems to be the state of the art in AI architectures, but purpose built neural network hardware might not actually benefit from the same architecture.
A neural network made from analog hardware could theoretically replace a significant portion of an LLM’s processing and not be limited by things like floating point precision or clock speeds. Who needs floating point when you can literally just multiply voltages together with a couple transistor junctions at the speed of light?
There is no way any breakthrough will be made with new technologies whilst the current AI speculation continues over RAM and GPUs, there seems to be still way to much money to be made in this bubble so most of capital resources in this area are being directed tô those “AI” initiatives.