llama.cpp with OpenAI chat completions API! 🦙
100% local. Powered by Metal!
*sound on*
In 2 steps:
1. brew install ggerganov/ggerganov/llama.cpp
2. llama-server --model -c 2048
P.S. All of this with a binary size of less than 5MB ;)
That's it! 🤗
100% local. Powered by Metal!
*sound on*
In 2 steps:
1. brew install ggerganov/ggerganov/llama.cpp
2. llama-server --model
P.S. All of this with a binary size of less than 5MB ;)
That's it! 🤗
Compatible with more than 7500+ GGUFs available on the Hugging Face Hub and more:
huggingface.co/models?library…
huggingface.co/models?library…
To poll the API all ya gotta do is:
curl --request POST \
--url http://localhost:8080/completion \
--header "Content-Type: application/json" \
--data '{"prompt": "The meaning of life is :", "n_predict": 512}'
curl --request POST \
--url http://localhost:8080/completion \
--header "Content-Type: application/json" \
--data '{"prompt": "The meaning of life is :", "n_predict": 512}'
Also works with streaming mode:
just add "stream": true
curl --request POST \
--url http://localhost:8080/completion \
--header "Content-Type: application/json" \
--data '{"prompt": "The meaning of life is :","n_predict": 512, "stream": true}'
just add "stream": true
curl --request POST \
--url http://localhost:8080/completion \
--header "Content-Type: application/json" \
--data '{"prompt": "The meaning of life is :","n_predict": 512, "stream": true}'
On Ollama & LMStudio:
https://twitter.com/reach_vb/status/1772749776699670772
• • •
Missing some Tweet in this thread? You can try to
force a refresh
