mirror of
https://github.com/hiyouga/LLaMA-Factory.git
synced 2025-08-03 20:22:49 +08:00
update vllm example
Former-commit-id: 49a2dfaf90982079239d116fcfac9ca72a8fc2b5
This commit is contained in:
parent
3b16f97c6c
commit
4b23159f53
@ -369,10 +369,11 @@ Use `python src/train_bash.py -h` to display arguments description.
|
||||
### Deploy with OpenAI-style API and vLLM
|
||||
|
||||
```bash
|
||||
CUDA_VISIBLE_DEVICES=0 API_PORT=8000 python src/api_demo.py \
|
||||
CUDA_VISIBLE_DEVICES=0,1 API_PORT=8000 python src/api_demo.py \
|
||||
--model_name_or_path mistralai/Mistral-7B-Instruct-v0.2 \
|
||||
--template mistral \
|
||||
--infer_backend vllm
|
||||
--infer_backend vllm \
|
||||
--vllm_enforce_eager
|
||||
```
|
||||
|
||||
### Use ModelScope Hub
|
||||
|
@ -369,10 +369,11 @@ docker compose -f ./docker-compose.yml up -d
|
||||
### 使用 OpenAI 风格 API 和 vLLM 部署
|
||||
|
||||
```bash
|
||||
CUDA_VISIBLE_DEVICES=0 API_PORT=8000 python src/api_demo.py \
|
||||
CUDA_VISIBLE_DEVICES=0,1 API_PORT=8000 python src/api_demo.py \
|
||||
--model_name_or_path mistralai/Mistral-7B-Instruct-v0.2 \
|
||||
--template mistral \
|
||||
--infer_backend vllm
|
||||
--infer_backend vllm \
|
||||
--vllm_enforce_eager
|
||||
```
|
||||
|
||||
### 使用魔搭社区
|
||||
|
Loading…
x
Reference in New Issue
Block a user