I won’t pretend to be an expert but my understanding is that the architecture loads experts from the SSD on demand with the RAM serving as a hot cache and the VRAM running the active experts.
As I said above I was able to run 70tok/s on my 4090w64GB ram and a samsung pcie4.0 ssd.
I haven’t run the benchmarks myself, but I can say the quality of response I was getting seemed on par with the frontier models I use for work (Claude opus/fable) and the published benchmarks for the model compete with those same models.
It is a q3 quantization so there is some small loss of accuracy, but from what I’ve understood those are modest losses in the worst case.
i would not recommend using the SSD to run local LLMS,its slow
best: VRAM only
good: VRAM + RAM
bad: Swapping
I don’t think they’re insinuating running inferences from the SSD. But you need somewhere to store the weights for the GPU to load.
i am trying to say: you need the SSD to store it but not swap to it due to its slowness
I won’t pretend to be an expert but my understanding is that the architecture loads experts from the SSD on demand with the RAM serving as a hot cache and the VRAM running the active experts.
As I said above I was able to run 70tok/s on my 4090w64GB ram and a samsung pcie4.0 ssd.
I haven’t run the benchmarks myself, but I can say the quality of response I was getting seemed on par with the frontier models I use for work (Claude opus/fable) and the published benchmarks for the model compete with those same models.
It is a q3 quantization so there is some small loss of accuracy, but from what I’ve understood those are modest losses in the worst case.