The main issue with Open / Local LLMs is the lack of broad support. It’s not hard for me to run Local LLMs, but apparently a decade of informal python experience and using linux mint is intimidating to some people. Apparently people want to click button and talk to phone god? Mostly speaks of a failing education system if you ask me.
I’m not even that good with AI, I just lurked a bit on !LocalLLaMA@sh.itjust.works and got mine running at like 30 tokens/second on my laptop.
I will have to defend users here and blame AI scientist/developers together with the whole docker culture. The way they package and delivery software is just terrible. And even though pytourch has support for all GPU apis the devs seems to have a hard time adding this support. Granted, part of that is also on pytourch that for some reason Don’t seem to be able to make a unified release that supports all GPUs and then you need different versions and by default IIRC they default package only supprt CUDA and CPU.
The main issue with Open / Local LLMs is the lack of broad support. It’s not hard for me to run Local LLMs, but apparently a decade of informal python experience and using linux mint is intimidating to some people. Apparently people want to click button and talk to phone god? Mostly speaks of a failing education system if you ask me.
I’m not even that good with AI, I just lurked a bit on !LocalLLaMA@sh.itjust.works and got mine running at like 30 tokens/second on my laptop.
I will have to defend users here and blame AI scientist/developers together with the whole docker culture. The way they package and delivery software is just terrible. And even though pytourch has support for all GPU apis the devs seems to have a hard time adding this support. Granted, part of that is also on pytourch that for some reason Don’t seem to be able to make a unified release that supports all GPUs and then you need different versions and by default IIRC they default package only supprt CUDA and CPU.