Suraj Subramanian
12602f32e2
Merge branch 'main' into subramen-patch-deadlinks
|
vor 8 Monaten | |
|---|---|---|
| .. | ||
| hf_text_generation_inference | vor 8 Monaten | |
| vllm | vor 8 Monaten | |
| README.md | vor 8 Monaten | |
| llama-on-prem.md | vor 8 Monaten | |
README.md
Running Llama2 On-Prem with vLLM and TGI
This tutorial shows how to use Llama 2 with vLLM and Hugging Face TGI to build Llama 2 on-prem apps.
* To run a quantized Llama2 model on iOS and Android, you can use the open source MLC LLM or llama.cpp. You can even make a Linux OS that boots to Llama2 (repo).