HswOAuth/llm_course
19
16
Fork 0
Code Issues 608 Pull Requests Actions Packages Projects Releases Wiki Activity

128G内存+24G英伟达GPU+CPU:13th i9 13-13900K 3.00GHz;能否跑llama 3.2 11B大模型 #219

New Issue
Open
opened 2024-10-17 21:28:00 +08:00 by 11608050661cs · 1 comment
11608050661cs commented 2024-10-17 21:28:00 +08:00
Copy Link

老师好，我的精务是医疗风险管理和医疗法律诉讼，需要使用相对大一些的模型。我的机子配置是128G内存+24G英伟达GPU+CPU:13th i9 13-13900K 3.00GHz。这样能否跑llama 3.2 11B大模型

老师好，我的精务是医疗风险管理和医疗法律诉讼，需要使用相对大一些的模型。我的机子配置是128G内存+24G英伟达GPU+CPU:13th i9 13-13900K 3.00GHz。这样能否跑llama 3.2 11B大模型
21970855250cs commented 2024-10-18 23:15:33 +08:00
Copy Link

可以的，就是显存有点吃力，模型加载使用bfloat16（如果GPU支持的话），大概估算就是1B参数需要2G显存；
另外还需要预留足够的显存用来做kv-cache，用来支持推理的时候输入的长上下文。

可以的，就是显存有点吃力，模型加载使用bfloat16（如果GPU支持的话），大概估算就是1B参数需要2G显存； 另外还需要预留足够的显存用来做kv-cache，用来支持推理的时候输入的长上下文。
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels   
bug

Something is not working

  
duplicate

This issue or pull request already exists

  
enhancement

New feature

  
help wanted

Need some help

  
invalid

Something is wrong

  
question

More information is needed

  
wontfix

This won't be fixed

No Label
bug
duplicate
enhancement
help wanted
invalid
question
wontfix
Milestone
Clear milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
No Assignees
2 Participants
Notifications
Due Date
The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: HswOAuth/llm_course#219
No description provided.
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?