As someone actually running LLMs locally for testing, unfortunately I’m not sure I agree.
For any passable sort of performance, you want as much as possible running on the GPU. Best bang for buck here is 1088 euros for a 24 GB RX 7900 XTX or 1681 for a 32 GB Radeon AI Pro R9700.
Now, you can fit many models on 24 GB, but they’re so far in output quality compared to frontier models, that they’re not actually good for this task. But add MoE offloading and 256 gigs of RAM and you can get a 4-bit quant of qwen 3.5 397B runnin. That’s about 3 grand for RAM. You’d then also need a decent CPU.
For even better performance, you can get a 256 GB Mac.
The upside is that you never run out of tokens. Even the damn 200$ plans for Claude and OpenAI have 5 hour limits that you can run into and then you have to wait again. The downside is that it won’t output fast enough to actually have to consider running out of tokens lol
I mean you’d use the 200 dollar tier if you keep running into usage limits because of Codex or something. There’s really no other reason for it IMHO.
At that point why not just run the model locally? 4 months of subscription would pay for a powerful enough setup.
As someone actually running LLMs locally for testing, unfortunately I’m not sure I agree.
For any passable sort of performance, you want as much as possible running on the GPU. Best bang for buck here is 1088 euros for a 24 GB RX 7900 XTX or 1681 for a 32 GB Radeon AI Pro R9700.
Now, you can fit many models on 24 GB, but they’re so far in output quality compared to frontier models, that they’re not actually good for this task. But add MoE offloading and 256 gigs of RAM and you can get a 4-bit quant of qwen 3.5 397B runnin. That’s about 3 grand for RAM. You’d then also need a decent CPU.
For even better performance, you can get a 256 GB Mac.
The upside is that you never run out of tokens. Even the damn 200$ plans for Claude and OpenAI have 5 hour limits that you can run into and then you have to wait again. The downside is that it won’t output fast enough to actually have to consider running out of tokens lol