• 14 Posts
  • 694 Comments
Joined 2 years ago
cake
Cake day: January 28th, 2025

help-circle


  • I am using a GLiNET ip kvm. Small, great software. Has HDMI and USB, can be powered from USB itself too.

    You can plug in files over the web interface like they are USB drives too. Can install tails ale and more plugins, can fake any monitor resolution you need…

    Very pleased for the price tag. Never let me down.














  • Yes i do.

    I esperiment a lot, but my stable setup is Qwen 3.8 27B Q6_K on my home server. Hardware is Ryzen 9 5950x with 64gb ddr4 and two old but effective NVIDIA RTX A4000 for a total of 32gb VRAM.

    Qwen 3.8 is the best model I can run in terms of output quality. It can perform all tasks pretty well and it’s reasonably fast at 500ts prefill and 30/40ts processing. There are much faster models but honestly not as good.

    I can also run gemma4 all on CPU, with no GPU at all.

    The key point is having both GPU on pci-e CPU, not on chip set bridge, so I am using a PCI riser 8x8x to share the 16x slot between the two GPUs. Inference speed is unaffected, but prefill benefits immensely.

    Software side, use llama.cpp on Linux, you get the best performance possible at the expense of a bit more complexity compared to other engines.

    From a power consumption perspective, I tined down my CPU governor and maximum frequency, and also the GPUs maximum watts. At ide the system is around 80w. At peak, it’s about 280w with the limiters, and inference speed doesn’t suffer too much. Without limiters I can easily reach 650w.

    Edit: I run llama-swap in front of llama-cpp, and openwebui as agent manager/ web GUI. I also have ComfyUI installed because why not but seldomly use it.

    You can read more of my setup on my wiki https://wiki.gardiol.org/8-llm-and-ai (no ads, no monetization whatsoever because fuck a hobby should not be a side gig)



  • that is … Debatable.

    Yeah even writing lead to cognitive loss, because now people don’t have to memorize everything.

    It’s just fear for the future and for change. And this is probably the most massive change in human history since fire was discovered.

    But don’t fear too much: the change is here and will stay, that’s the only sure thing

    Edit: what I mean is that accepting the fact AI is here and won’t go away because we don’t like it is the path to start regulating it at government level and agreeing on ethical rules to use it. Just saying you are a bad person of you use it it’s like hiding under the sand. You lose.