Matthias Reso 8c45a713ee Adapt example documentation		před 1 rokem
..
chat_completion	3ddf755539 Move examples into subfolders	před 1 rokem
code_llama	ccda6fb8ca Move inference scripts into example folder	před 1 rokem
hf_text_generation_inference	ccda6fb8ca Move inference scripts into example folder	před 1 rokem
vllm	3ddf755539 Move examples into subfolders	před 1 rokem
README.md	8c45a713ee Adapt example documentation	před 1 rokem
inference.py	ccda6fb8ca Move inference scripts into example folder	před 1 rokem
samsum_prompt.txt	ccda6fb8ca Move inference scripts into example folder	před 1 rokem

Inference

This folder contains inference examples for Llama 2. So far, we have provided support for three methods of inference:

inference script script provides support for Hugging Face accelerate, PEFT and FSDP fine tuned models.
vllm/inference.py script takes advantage of vLLM's paged attention concept for low latency.
The hf_text_generation_inference folder contains information on Hugging Face Text Generation Inference (TGI).

For more in depth information on inference including inference safety checks and examples, see the inference documentation here.