Inicio Explorar Axuda
Rexistro Iniciar sesión
marx
/
llama-recipes
1
0
Fork 0
Ficheiros Incidencias 0 Pull Requests 0 Wiki
Árbore: e69fd49c4f
Ramas Etiquetas
llama-recipes
/
inference
Hamid Shojanazeri e69fd49c4f moved update to a separate file hai 1 ano
..
hf-text-generation-inference b3067b55dc fix typos and spelling errors hai 1 ano
README.md e69fd49c4f moved update to a separate file hai 1 ano
chat_completion.py 557e881fcc aliginng the pad token with HF latest hai 1 ano
chat_utils.py f6152893d8 update prompts hai 1 ano
chats.json f6152893d8 update prompts hai 1 ano
checkpoint_converter_fsdp_hf.py 50e9d17045 add the default option for find the HF model_name/path from train_param.yaml hai 1 ano
inference.py c4e96af6ee clean up hai 1 ano
model_utils.py 76a187c4d2 clean up hai 1 ano
safety_utils.py 4767f09ecd Initial commit hai 1 ano
samsum_prompt.txt 4767f09ecd Initial commit hai 1 ano
vLLM_inference.py 4767f09ecd Initial commit hai 1 ano

README.md

Inference

This folder contains inference examples for Llama 2. So far, we have provided support for three methods of inference:

  1. inference script script provides support for Hugging Face accelerate, PEFT and FSDP fine tuned models.

  2. vLLM_inference.py script takes advantage of vLLM's paged attention concept for low latency.

  3. The hf-text-generation-inference folder contains information on Hugging Face Text Generation Inference (TGI).

For more in depth information on inference including inference safety checks and examples, see the inference documentation here.