【求助帖】22-25.2.18-deepseek理论介绍请教一下如何使用vllm连接本地模型 #586

New Issue

CharlieXie · 2025-02-26T21:45:54+08:00

CharlieXie commented

2025-02-26 21:45:54 +08:00

使用如下代码:
from vllm import LLM, SamplingParams

tp_size = 4 # Tensor Parallelism
sampling_params = SamplingParams(temperature=0.7, top_p=0.9, max_tokens=100)
model_name = "deepseek-ai/deepseek-llm-67b-base"
llm = LLM(model=model_name, trust_remote_code=True, gpu_memory_utilization=0.9, tensor_parallel_size=tp_size)

prompts = [
"If everyone in a country loves one another,",
"The research should also focus on the technologies",
"To determine if the label is correct, we need to"
]
outputs = llm.generate(prompts, sampling_params)

generated_text = [output.outputs[0].text for output in outputs]
print(generated_text)

本地模型路径：
/root/autodl-tmp/model

使用如下代码: from vllm import LLM, SamplingParams tp_size = 4 # Tensor Parallelism sampling_params = SamplingParams(temperature=0.7, top_p=0.9, max_tokens=100) model_name = "deepseek-ai/deepseek-llm-67b-base" llm = LLM(model=model_name, trust_remote_code=True, gpu_memory_utilization=0.9, tensor_parallel_size=tp_size) prompts = [ "If everyone in a country loves one another,", "The research should also focus on the technologies", "To determine if the label is correct, we need to" ] outputs = llm.generate(prompts, sampling_params) generated_text = [output.outputs[0].text for output in outputs] print(generated_text) 本地模型路径： /root/autodl-tmp/model <img width="373" alt="image" src="/attachments/6a133d8b-d3e8-4d20-89fb-12d51246e158">

image.png

22 KiB

CharlieXie closed this issue

2025-02-27 23:08:32 +08:00

Sign in to join this conversation.