Inicio Explorar Axuda
Rexistro Iniciar sesión
marx
/
llama-recipes
1
0
Fork 0
Ficheiros Incidencias 0 Pull Requests 0 Wiki
Árbore: 3ddf755539
Ramas Etiquetas
llama-recipes
/
examples
Matthias Reso 3ddf755539 Move examples into subfolders hai 1 ano
..
chat_completion 3ddf755539 Move examples into subfolders hai 1 ano
code_llama ccda6fb8ca Move inference scripts into example folder hai 1 ano
hf_text_generation_inference ccda6fb8ca Move inference scripts into example folder hai 1 ano
vllm 3ddf755539 Move examples into subfolders hai 1 ano
README.md ccda6fb8ca Move inference scripts into example folder hai 1 ano
inference.py ccda6fb8ca Move inference scripts into example folder hai 1 ano
samsum_prompt.txt ccda6fb8ca Move inference scripts into example folder hai 1 ano

README.md

Inference

This folder contains inference examples for Llama 2. So far, we have provided support for three methods of inference:

  1. inference script script provides support for Hugging Face accelerate, PEFT and FSDP fine tuned models.

  2. vLLM_inference.py script takes advantage of vLLM's paged attention concept for low latency.

  3. The hf_text_generation_inference folder contains information on Hugging Face Text Generation Inference (TGI).

For more in depth information on inference including inference safety checks and examples, see the inference documentation here.