Definitely. We do this at my job. But a server big enough to provide a comparable service would cost at least $30k. And you likely need more than one. And you still need someone to maintain it.
I’ve heard that GPU’s for this work are exceedingly hard to get ahold of as well. Glm 5, which i think is near frontier model capabilities, takes around 1.5 Tb of vram un-quantized.
Glm 5, which i think is near frontier model capabilities, takes around 1.5 Tb of vram un-quantized.
DeepSeek’s latest version is 508GB, and even that is wasteful, compared to the wild difference in sizes with some of the quantized versions. The huge amount of parameters and weight sizes are completely unoptimized in terms of sizes, and there’s all kinds of diminishing returns with the haphazardous way these LLM corps are applying new model versions.
Definitely. We do this at my job. But a server big enough to provide a comparable service would cost at least $30k. And you likely need more than one. And you still need someone to maintain it.
I’ve heard that GPU’s for this work are exceedingly hard to get ahold of as well. Glm 5, which i think is near frontier model capabilities, takes around 1.5 Tb of vram un-quantized.
DeepSeek’s latest version is 508GB, and even that is wasteful, compared to the wild difference in sizes with some of the quantized versions. The huge amount of parameters and weight sizes are completely unoptimized in terms of sizes, and there’s all kinds of diminishing returns with the haphazardous way these LLM corps are applying new model versions.
Not hard. Just really fucking expensive and might have a long lead time.
So hard, then.