HswOAuth/llm_course
19
16
Fork 0
Code Issues 608 Pull Requests Actions Packages Projects Releases Wiki Activity

Chinese-LLaMA-Alpaca-3本地构建API可调用，但NextChat无法正常调用 #206

New Issue
Open
opened 2024-10-17 10:36:19 +08:00 by 11153343824cs · 4 comments
11153343824cs commented 2024-10-17 10:36:19 +08:00
Copy Link

是软件里配置错误了吗

我的接口地址是这个：http://172.18.56.62:19327/v1/completions

是软件里配置错误了吗 我的接口地址是这个：http://172.18.56.62:19327/v1/completions
image-20241017001839356.png
88 KiB
image-20241017001849641.png
75 KiB
image-20241017001830610.png
374 KiB
image-20241017001839356.png image-20241017001849641.png image-20241017001830610.png
11153343824cs started working 2024-10-17 10:36:38 +08:00
best_newnew commented 2024-10-17 14:39:43 +08:00
Copy Link

本地构建API可调用是什么意思呀？
我看你图上的操作都很对欸，API接口设置，包括在后面的LLAMA-CHINESE模型选择
可以提供更多的信息吗，我想看看你模型部署时候的输出，具体是在“python openai_api_server.py --gpus 0 --base_model /home/ganjialing/.cache/modelscope/hub/ChineseAlpacaGroup/llama-3-chinese-8b-instruct-v3”这句代码后的输出。

或者你可以对照一下这篇帖子里的步骤，写得满详细的，看看是不是有问题
llama3仿openai api实验
不知道为啥设置不了链接，是第#173号帖子

如果自己解决了可以告诉我一下出的什么问题嘛，贴在这篇帖子后面就好啦

本地构建API可调用是什么意思呀？ 我看你图上的操作都很对欸，API接口设置，包括在后面的LLAMA-CHINESE模型选择 可以提供更多的信息吗，我想看看你模型部署时候的输出，具体是在“python openai_api_server.py --gpus 0 --base_model /home/ganjialing/.cache/modelscope/hub/ChineseAlpacaGroup/llama-3-chinese-8b-instruct-v3”这句代码后的输出。 或者你可以对照一下这篇帖子里的步骤，写得满详细的，看看是不是有问题 [llama3仿openai api实验](HswOAuth/llm_course#173) 不知道为啥设置不了链接，是第#173号帖子 如果自己解决了可以告诉我一下出的什么问题嘛，贴在这篇帖子后面就好啦
11153343824cs commented 2024-10-17 17:16:22 +08:00
Author
Copy Link

llama3仿openai api实验, 提示我没权限访问这个链接

这个是我运行后的完整输出

(chinese_llama_alpaca_3) shiyl@LAPTOP-8VC9E9I1:/mnt/d/03.CodeProjects/LLM/model/Chinese-LLaMA-Alpaca-3$ python scripts/oai_api_demo/openai_api_server.py \
--load_in_4bit \
--base_model /home/shiyl/.cache/modelscope/hub/ChineseAlpacaGroup/llama-3-chinese-8b-instruct-v3 \
--gpus 0
Using device: cuda:0
Loading checkpoint shards: 100%|██████████████████████████████████████████████████████████| 4/4 [00:11<00:00,  2.96s/it]
Vocab of the base model: 128256
Vocab of the tokenizer: 128256
2024-10-17 17:15:15,634 - INFO - Started server process [635]
2024-10-17 17:15:15,634 - INFO - Waiting for application startup.
2024-10-17 17:15:15,634 - INFO - Application startup complete.
2024-10-17 17:15:15,634 - INFO - Uvicorn running on http://0.0.0.0:19327 (Press CTRL+C to quit)

您看下。

[llama3仿openai api实验](HswOAuth/llm_course#173), 提示我没权限访问这个链接 这个是我运行后的完整输出 ```shell (chinese_llama_alpaca_3) shiyl@LAPTOP-8VC9E9I1:/mnt/d/03.CodeProjects/LLM/model/Chinese-LLaMA-Alpaca-3$ python scripts/oai_api_demo/openai_api_server.py \ --load_in_4bit \ --base_model /home/shiyl/.cache/modelscope/hub/ChineseAlpacaGroup/llama-3-chinese-8b-instruct-v3 \ --gpus 0 Using device: cuda:0 Loading checkpoint shards: 100%|██████████████████████████████████████████████████████████| 4/4 [00:11<00:00, 2.96s/it] Vocab of the base model: 128256 Vocab of the tokenizer: 128256 2024-10-17 17:15:15,634 - INFO - Started server process [635] 2024-10-17 17:15:15,634 - INFO - Waiting for application startup. 2024-10-17 17:15:15,634 - INFO - Application startup complete. 2024-10-17 17:15:15,634 - INFO - Uvicorn running on http://0.0.0.0:19327 (Press CTRL+C to quit) ``` 您看下。
12390900721cs commented 2024-10-17 18:30:45 +08:00
Copy Link

1. 这里很明显你的端口号是19327，这个是没问题哒~

但是请问一下你的实验是在本地电脑做的还是用的服务器呢？如果是远程服务器的话您的API接口地址设置为172.18.56.62:19327是没问题哒，但是如果就是用的自己的电脑的话应该设置为http://localhost:19327或者http://0.0.0.0:19327噢~

2. 没有权限访问这个链接

我不知道为什么我自己写的自己都没权限访问，不过你可以在汇视威论坛上搜#173会自动出来的~

## 1. 这里很明显你的端口号是19327，这个是没问题哒~ 但是请问一下你的实验是在本地电脑做的还是用的服务器呢？如果是远程服务器的话您的API接口地址设置为172.18.56.62:19327是没问题哒，但是如果就是用的自己的电脑的话应该设置为http://localhost:19327或者http://0.0.0.0:19327噢~ ## 2. 没有权限访问这个链接 我不知道为什么我自己写的自己都没权限访问，不过你可以在汇视威论坛上搜#173会自动出来的~
11153343824cs stopped working 2024-10-17 18:43:25 +08:00
8 hours 6 minutes
11153343824cs started working 2024-10-17 18:43:37 +08:00
11153343824cs commented 2024-10-17 18:51:20 +08:00
Author
Copy Link

找到问题了，和llama3仿openai api实验一样
通过这个命令解决了

pip install pydantic==1.10.11 -i https://pypi.tuna.tsinghua.edu.cn/simple


(chinese_llama_alpaca_3) shiyl@LAPTOP-8VC9E9I1:/mnt/d/03.CodeProjects/LLM/model/Chinese-LLaMA-Alpaca-3$ pip install pydantic==1.10.11 -i https://pypi.tuna.tsinghua.edu.cn/simple
Looking in indexes: https://pypi.tuna.tsinghua.edu.cn/simple
Collecting pydantic==1.10.11
  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/4e/d7/04c964cda3b5e2c05d40d89fffa4449512c757476c9dd6aee9154a2e2603/pydantic-1.10.11-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (3.2 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 3.2/3.2 MB 26.3 MB/s eta 0:00:00
Requirement already satisfied: typing-extensions>=4.2.0 in /home/shiyl/miniconda3/envs/chinese_llama_alpaca_3/lib/python3.8/site-packages (from pydantic==1.10.11) (4.12.2)
Installing collected packages: pydantic
  Attempting uninstall: pydantic
    Found existing installation: pydantic 2.9.2
    Uninstalling pydantic-2.9.2:
      Successfully uninstalled pydantic-2.9.2
Successfully installed pydantic-1.10.11

更换版本后需要杀掉已经启动的服务进程，再启动新的服务

ps -ef | grep python | grep -v grep | grep oai_api_demo |cut -c 8-17 | xargs kill -9

python scripts/oai_api_demo/openai_api_server.py \
--load_in_4bit \
--base_model /home/shiyl/.cache/modelscope/hub/ChineseAlpacaGroup/llama-3-chinese-8b-instruct-v3 \
--gpus 0

经过这些过程就可以了

image

找到问题了，和[llama3仿openai api实验](HswOAuth/llm_course#173)一样 通过这个命令解决了 ```shell pip install pydantic==1.10.11 -i https://pypi.tuna.tsinghua.edu.cn/simple (chinese_llama_alpaca_3) shiyl@LAPTOP-8VC9E9I1:/mnt/d/03.CodeProjects/LLM/model/Chinese-LLaMA-Alpaca-3$ pip install pydantic==1.10.11 -i https://pypi.tuna.tsinghua.edu.cn/simple Looking in indexes: https://pypi.tuna.tsinghua.edu.cn/simple Collecting pydantic==1.10.11 Downloading https://pypi.tuna.tsinghua.edu.cn/packages/4e/d7/04c964cda3b5e2c05d40d89fffa4449512c757476c9dd6aee9154a2e2603/pydantic-1.10.11-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (3.2 MB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 3.2/3.2 MB 26.3 MB/s eta 0:00:00 Requirement already satisfied: typing-extensions>=4.2.0 in /home/shiyl/miniconda3/envs/chinese_llama_alpaca_3/lib/python3.8/site-packages (from pydantic==1.10.11) (4.12.2) Installing collected packages: pydantic Attempting uninstall: pydantic Found existing installation: pydantic 2.9.2 Uninstalling pydantic-2.9.2: Successfully uninstalled pydantic-2.9.2 Successfully installed pydantic-1.10.11 ``` 更换版本后需要杀掉已经启动的服务进程，再启动新的服务 ```shell ps -ef | grep python | grep -v grep | grep oai_api_demo |cut -c 8-17 | xargs kill -9 python scripts/oai_api_demo/openai_api_server.py \ --load_in_4bit \ --base_model /home/shiyl/.cache/modelscope/hub/ChineseAlpacaGroup/llama-3-chinese-8b-instruct-v3 \ --gpus 0 ``` 经过这些过程就可以了 ![image](/attachments/37a26f24-735b-474a-9a0f-ed29d7954ee8)
image.png
113 KiB
11153343824cs stopped working 2024-10-18 00:51:15 +08:00
6 hours 7 minutes
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels   
bug

Something is not working

  
duplicate

This issue or pull request already exists

  
enhancement

New feature

  
help wanted

Need some help

  
invalid

Something is wrong

  
question

More information is needed

  
wontfix

This won't be fixed

No Label
bug
duplicate
enhancement
help wanted
invalid
question
wontfix
Milestone
Clear milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
No Assignees
3 Participants
Notifications
Total Time Spent: 14 hours 14 minutes
11153343824cs
14 hours 14 minutes
Due Date
The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: HswOAuth/llm_course#206
No description provided.
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?