HswOAuth/llm_course
19
16
Fork 0
Code Issues 608 Pull Requests Actions Packages Projects Releases Wiki Activity

进行RLHF微调中的奖励模型训练阶段，在进行数据处理时出现错误，在本地电脑能成功跑通，但是显存资源不够。在汇视威平台就出现了以下的报错信息。本地的和汇视威平台的transformers库以及tokenizers库都相同，不明白在汇视威平台执行图1的代码时为什么报错？求助如何解决？ #167

New Issue
Closed
opened 2024-10-09 14:13:48 +08:00 by XPH · 2 comments
21970855250cs commented 2024-10-09 20:53:18 +08:00
Copy Link

这里应该是中文导致的错误，tokenzier有两个实现，这里应该是rust的报错。可以在初始化tokenizer的时候将use_fast设置成false试一下。

这里应该是中文导致的错误，tokenzier有两个实现，这里应该是rust的报错。可以在初始化tokenizer的时候将use_fast设置成false试一下。
XPH commented 2024-10-09 21:13:33 +08:00
Author
Copy Link

这里应该是中文导致的错误，tokenzier有两个实现，这里应该是rust的报错。可以在初始化tokenizer的时候将use_fast设置成false试一下。

还是不行，我加载的tokenizer是glm4-9b-chat

> 这里应该是中文导致的错误，tokenzier有两个实现，这里应该是rust的报错。可以在初始化tokenizer的时候将use_fast设置成false试一下。 还是不行，我加载的tokenizer是glm4-9b-chat
XPH closed this issue 2024-10-11 09:10:37 +08:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels   
bug

Something is not working

  
duplicate

This issue or pull request already exists

  
enhancement

New feature

  
help wanted

Need some help

  
invalid

Something is wrong

  
question

More information is needed

  
wontfix

This won't be fixed

No Label
bug
duplicate
enhancement
help wanted
invalid
question
wontfix
Milestone
Clear milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
No Assignees
2 Participants
Notifications
Due Date
The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: HswOAuth/llm_course#167
No description provided.
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?