Memory Prices Indefinitely High: SK Hynix Says Shortage Continues Through 2030 We haven't been complaining about RAM prices quite as much on the podcast
Self hosting also requires a ton of RAM because each individual user has to load a (usually slightly smaller) model + KV cache, whereas for hosted, they load the huge-ass model once (per cluster) and then store KV caches for a bunch of users, which has better scaling.
The memory companies win whether you self-host or pay a subscription. They might win more if you self-host because look at the cost of self-hosting, it’s actually pretty steep if you don’t have the hardware already from the before times.
Self hosting also requires a ton of RAM because each individual user has to load a (usually slightly smaller) model + KV cache, whereas for hosted, they load the huge-ass model once (per cluster) and then store KV caches for a bunch of users, which has better scaling.
The memory companies win whether you self-host or pay a subscription. They might win more if you self-host because look at the cost of self-hosting, it’s actually pretty steep if you don’t have the hardware already from the before times.