10-基于vLLM本地模型部署 #201

New Issue

Open

opened 2024-10-16 17:27:18 +08:00 by 12535224197cs · 0 comments

12535224197cs commented 2024-10-16 17:27:18 +08:00

Copy Link

1. 创建环境

2. 单卡推理

python vllm_standalone.py

3. 基于ray做分布式多卡推理

3.1 启动ray

# 启动ray
HIP_VISIBLE_DEVICES=0,1,2,3 ray start --head --dashboard-host 0.0.0.0 --num-gpus=4
# 查看ray集群状态：
ray status

3.2 分布式推理

python vllm_distributed_ray.py

3.3 使用ray查看任务

4. 启动多卡推理

# 部署多卡服务
python -m vllm.entrypoints.openai.api_server --model /dataset/Qwen1.5-0.5B-Chat/ --tensor-parallel-size 4 --dtype float16
# 使用OpenAI python SDK调用vllm部署的模型
python vllm_openai_call.py

5.部署Qwen2-7B-Instruct

# 使用如下命令可以部署一个Qwen2-7B-Instruct模型，并模拟openai api对外提供服务
python -m vllm.entrypoints.openai.api_server \
  --model /dataset/Qwen2-7B-Instruct/ \
  --served-model-name Qwen2-7B-Instruct \
  --tensor-parallel-size 4 \
  --dtype float16
# tools使用
python /code/ex04_api_call_tools.py

## 1. 创建环境  ## 2. 单卡推理 `python vllm_standalone.py`  ## 3. 基于ray做分布式多卡推理 ### 3.1 启动ray ``` # 启动ray HIP_VISIBLE_DEVICES=0,1,2,3 ray start --head --dashboard-host 0.0.0.0 --num-gpus=4 # 查看ray集群状态： ray status ```  ### 3.2 分布式推理 python vllm_distributed_ray.py <video src="/attachments/d2245e6b-8ec7-40ed-82ec-f52796f179ce" title="2基于ray做分布式推理.mov" controls></video> ### 3.3 使用ray查看任务   ## 4. 启动多卡推理 ``` # 部署多卡服务 python -m vllm.entrypoints.openai.api_server --model /dataset/Qwen1.5-0.5B-Chat/ --tensor-parallel-size 4 --dtype float16 # 使用OpenAI python SDK调用vllm部署的模型 python vllm_openai_call.py ```  ## 5.部署Qwen2-7B-Instruct ``` # 使用如下命令可以部署一个Qwen2-7B-Instruct模型，并模拟openai api对外提供服务 python -m vllm.entrypoints.openai.api_server \ --model /dataset/Qwen2-7B-Instruct/ \ --served-model-name Qwen2-7B-Instruct \ --tensor-parallel-size 4 \ --dtype float16 # tools使用 python /code/ex04_api_call_tools.py ``` <video src="/attachments/5471b07b-d23b-4f58-b09c-453f48abbe1a" title="5.mov" controls></video>

5.mov

37 MiB

2基于ray做分布式推理.mov

108 MiB

Sign in to join this conversation.

No Branch/Tag Specified

Branches Tags

main

No results found.

Labels

Clear labels   

bug

Something is not working

  

duplicate

This issue or pull request already exists

  

enhancement

New feature

  

help wanted

Need some help

  

invalid

Something is wrong

  

question

More information is needed

  

wontfix

This won't be fixed

No Label

bug

duplicate

enhancement

help wanted

invalid

question

wontfix

Milestone

Clear milestone

No items

No Milestone

Projects

Clear projects

No project

Assignees

Clear assignees

No Assignees

1 Participants

Notifications

Subscribe

Due Date

The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: HswOAuth/llm_course#201

No description provided.