I have avoided AI (LLMS) because it didn’t seem nearly useful enough to warrant supporting these tech assholes. Is there anyway to use it locally on a normal device (16 Gb ram, 8 year old Geforce graphics, etc?). Can I just slow it down and get it to work locally?
Shittiest local inference engine. Use llama.cpp instead.
Old version 3 model from Alibaba. 4 billion parameters total. It will struggle with anything other than natural language generation.
Running on a 6 vcore VM with no GPU pass through.
Time.To.First.Token. How long it takes the model to start responding after you hit enter. Dependent on not only processing speed but also context length.
Cold start = no cache = slow. But faster when mid conversation and the key-value cache is hot.
100 tokens per second. 1 token is roughly 1/4 word.