Home Explore Help
Register Sign In
marx
/
llama-recipes
1
0
Fork 0
Files Issues 0 Pull Requests 0 Wiki
Tree: ccda6fb8ca
Branches Tags
llama-recipes
/
examples
Matthias Reso ccda6fb8ca Move inference scripts into example folder 1 year ago
..
code_llama ccda6fb8ca Move inference scripts into example folder 1 year ago
hf_text_generation_inference ccda6fb8ca Move inference scripts into example folder 1 year ago
README.md ccda6fb8ca Move inference scripts into example folder 1 year ago
chat_completion.py ccda6fb8ca Move inference scripts into example folder 1 year ago
chats.json ccda6fb8ca Move inference scripts into example folder 1 year ago
inference.py ccda6fb8ca Move inference scripts into example folder 1 year ago
samsum_prompt.txt ccda6fb8ca Move inference scripts into example folder 1 year ago
vLLM_inference.py ccda6fb8ca Move inference scripts into example folder 1 year ago

README.md

Inference

This folder contains inference examples for Llama 2. So far, we have provided support for three methods of inference:

  1. inference script script provides support for Hugging Face accelerate, PEFT and FSDP fine tuned models.

  2. vLLM_inference.py script takes advantage of vLLM's paged attention concept for low latency.

  3. The hf_text_generation_inference folder contains information on Hugging Face Text Generation Inference (TGI).

For more in depth information on inference including inference safety checks and examples, see the inference documentation here.