• OhneHose@feddit.org
    link
    fedilink
    English
    arrow-up
    5
    ·
    21 hours ago

    Judging from the size of other frontier models on hf, you won’t host it.

    The best/biggest you can squeeze on 64gb ram rn is ~70-80B in a Q4 Quant. Going lower with quantization will usually leave you with notable degraded performance.

      • OhneHose@feddit.org
        link
        fedilink
        English
        arrow-up
        2
        ·
        13 hours ago

        Depends on your hardware. Apples unified memory system can give near complete access to the ram for their GPU/npu.

        Apart from that you don’t need a single bit of vram/gpu to host a model. Then can be run completely on CPU+ram. But that comes with a significant processing penalty.

        On top of that there’s new “engines” which can run a LLM from disk storage +system ram/vram, yet again with lots of speed penalities, since you chunk the model into small pieces and load them into ram accordingly. This will also cripple the lifetime of ssds with the sheer amount of read/write cycles.

        Then there’s also dedicated systems or extension cards which can be used as accelerator.

        None the less we speak about a minimum of 5000€'s for a system which can run llms and (lightweight) games simultaneously.

      • B0rax@feddit.org
        link
        fedilink
        English
        arrow-up
        5
        ·
        18 hours ago

        You can also get systems with unified memory (from amd for example, or from Apple), which will be available* up to 512gb of ram.

        *that means, if you pay the equivalent of a middle sized new car.