423A35C7/LLaMA-Factory
1
0
Fork 0
mirror of https://github.com/hiyouga/LLaMA-Factory.git synced 2025-08-27 00:02:51 +08:00
Code Issues Packages Projects Releases Wiki Activity

update vllm example

Browse Source 
Former-commit-id: 49a2dfaf90982079239d116fcfac9ca72a8fc2b5
This commit is contained in:
hiyouga 2024-04-02 22:45:20 +08:00
parent 3b16f97c6c
commit 4b23159f53
2 changed files with 6 additions and 4 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

5
README.md
View File

@ -369,10 +369,11 @@ Use `python src/train_bash.py -h` to display arguments description.
### Deploy with OpenAI-style API and vLLM
```bash
CUDA_VISIBLE_DEVICES=0 API_PORT=8000 python src/api_demo.py \
CUDA_VISIBLE_DEVICES=0,1 API_PORT=8000 python src/api_demo.py \
--model_name_or_path mistralai/Mistral-7B-Instruct-v0.2 \
--template mistral \
--infer_backend vllm
--infer_backend vllm \
--vllm_enforce_eager
```
### Use ModelScope Hub

5
README_zh.md
View File

@ -369,10 +369,11 @@ docker compose -f ./docker-compose.yml up -d
### 使用 OpenAI 风格 API 和 vLLM 部署
```bash
CUDA_VISIBLE_DEVICES=0 API_PORT=8000 python src/api_demo.py \
CUDA_VISIBLE_DEVICES=0,1 API_PORT=8000 python src/api_demo.py \
--model_name_or_path mistralai/Mistral-7B-Instruct-v0.2 \
--template mistral \
--infer_backend vllm
--infer_backend vllm \
--vllm_enforce_eager
```
### 使用魔搭社区