Hamid Shojanazeri 85a4ed1b65 Merge branch 'main' into update_paddings		1 年之前
..
hf-text-generation-inference	b3067b55dc fix typos and spelling errors	1 年之前
README.md	3c84962685 adding checkpoint conversion	1 年之前
chat_completion.py	51269b816f moving Bt to the try block	1 年之前
chat_utils.py	f6152893d8 update prompts	1 年之前
chats.json	f6152893d8 update prompts	1 年之前
checkpoint_converter_fsdp_hf.py	50e9d17045 add the default option for find the HF model_name/path from train_param.yaml	1 年之前
inference.py	85a4ed1b65 Merge branch 'main' into update_paddings	1 年之前
model_utils.py	76a187c4d2 clean up	1 年之前
safety_utils.py	754b5d22c7 clean up and typo fixes	1 年之前
samsum_prompt.txt	4767f09ecd Initial commit	1 年之前
vLLM_inference.py	4767f09ecd Initial commit	1 年之前

Inference

This folder contains inference examples for Llama 2. So far, we have provided support for three methods of inference:

inference script script provides support for Hugging Face accelerate, PEFT and FSDP fine tuned models.
vLLM_inference.py script takes advantage of vLLM's paged attention concept for low latency.
The hf-text-generation-inference folder contains information on Hugging Face Text Generation Inference (TGI).

For more in depth information on inference including inference safety checks and examples, see the inference documentation here.