HswOAuth/llm_course
19
16
Fork 0
Code Issues 608 Pull Requests Actions Packages Projects Releases Wiki Activity

4机16卡训练实验报告 #67

New Issue
Closed
opened 2024-09-12 12:38:38 +08:00 by 21547230244cs · 0 comments
21547230244cs commented 2024-09-12 12:38:38 +08:00
Copy Link
  1. 首先，先在平台创立四个notebook（因为目标是训练4机16卡的实验）

  1. 在四个notebook/四台机器中分别都需要设置环境变量

设置环境变量

  1. 用ifconfig来查看节点的IP地址，选中一台机器/服务器节点为主节点。这里我选中的是IP地址为10.244.132.125作为我的主节点

  1. 然后在四台机器上分别输入四个对应的代码

NODE_RANK = 0 对应第一台机器

NODE_RANK=1 对应第二台机器

NODE_RANK = 2 对应第三台机器

NODE_RANK= 3 对应第四台机器

  1. 模型训练过程展示

  1. 等待模型跑完训练后，然后将lora模型转换并合并

先将path改为hf形式

保存训练出来的数据

  1. 跑完模型后，在主节点多开几个终端进行模型部署调用。在每个终端里分别输入以下代码

图1

图2

图3

图4

  1. 结果

1. 首先，先在平台创立四个notebook（因为目标是训练4机16卡的实验） ![](https://cdn.nlark.com/yuque/0/2024/png/48118617/1726113253530-5468de7c-54c9-4e45-9059-a3677d5c912d.png) ![](https://cdn.nlark.com/yuque/0/2024/png/48118617/1726113253808-88af1dcd-f3a5-4ec5-8e63-040b4dcd68ed.png) 2. 在四个notebook/四台机器中分别都需要设置环境变量 ![设置环境变量](https://cdn.nlark.com/yuque/0/2024/png/48118617/1726113254061-b57de7a0-e338-4df5-ba6e-da109f191024.png) 3. 用ifconfig来查看节点的IP地址，选中一台机器/服务器节点为主节点。这里我选中的是IP地址为10.244.132.125作为我的主节点 ![](https://cdn.nlark.com/yuque/0/2024/png/48118617/1726113254284-36e6e58b-eaa4-45a9-b06a-b0c98e6336d5.png) 4. 然后在四台机器上分别输入四个对应的代码 ![NODE_RANK = 0 对应第一台机器](https://cdn.nlark.com/yuque/0/2024/png/48118617/1726113254534-dff652fb-e0ca-45e4-9272-71a612750378.png) ![NODE_RANK=1 对应第二台机器](https://cdn.nlark.com/yuque/0/2024/png/48118617/1726113254726-f16528fb-235c-49cd-b642-352d8b965757.png) ![NODE_RANK = 2 对应第三台机器](https://cdn.nlark.com/yuque/0/2024/png/48118617/1726113254907-4c11ceee-f37b-48b4-aba4-8fbf92085198.png) ![NODE_RANK= 3 对应第四台机器](https://cdn.nlark.com/yuque/0/2024/png/48118617/1726113255089-24e77e7d-867f-40a6-aad9-35ff95cdc1ea.png) 5. 模型训练过程展示 ![](https://cdn.nlark.com/yuque/0/2024/png/48118617/1726113255396-5fdf3397-f863-4be7-b1a0-add84df2f236.png) 6. 等待模型跑完训练后，然后将lora模型转换并合并 ![先将path改为hf形式](https://cdn.nlark.com/yuque/0/2024/png/48118617/1726113255648-b06b8467-5319-4261-a423-aba999512b7f.png) ![保存训练出来的数据](https://cdn.nlark.com/yuque/0/2024/png/48118617/1726113255976-3837b353-247a-42b6-be72-49d7b8ea0305.png) 7. 跑完模型后，在主节点多开几个终端进行模型部署调用。在每个终端里分别输入以下代码 ![图1](https://cdn.nlark.com/yuque/0/2024/png/48118617/1726113256322-2b9533b5-515f-49b4-93e5-abfc6e91d4a6.png) ![图2](https://cdn.nlark.com/yuque/0/2024/png/48118617/1726113256738-15c9c150-e8b0-4b8a-8cd1-df044a2bff46.png) ![图3](https://cdn.nlark.com/yuque/0/2024/png/48118617/1726113257195-e870f42d-c167-4723-b177-bbd1e618faff.png) ![图4](https://cdn.nlark.com/yuque/0/2024/png/48118617/1726113257563-94bb6360-910e-42b3-ab86-e709ffaee8ec.png) 8. 结果 ![](https://cdn.nlark.com/yuque/0/2024/png/48118617/1726113257956-3e3f7458-5e24-4c0d-8e80-3c728efb3568.png)
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels   
bug

Something is not working

  
duplicate

This issue or pull request already exists

  
enhancement

New feature

  
help wanted

Need some help

  
invalid

Something is wrong

  
question

More information is needed

  
wontfix

This won't be fixed

No Label
bug
duplicate
enhancement
help wanted
invalid
question
wontfix
Milestone
Clear milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
No Assignees
1 Participants
Notifications
Due Date
The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: HswOAuth/llm_course#67
No description provided.
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?