HswOAuth/llm_course
19
16
Fork 0
Code Issues 608 Pull Requests Actions Packages Projects Releases Wiki Activity

关于重复主题的新闻去重 #448

New Issue
Open
opened 2024-12-26 22:33:49 +08:00 by llm · 1 comment
llm commented 2024-12-26 22:33:49 +08:00
Copy Link

请问各位老师，我有一些csv文件的新闻文本，同一件事由于不同的人编写导致差异较大，如何进行去重

请问各位老师，我有一些csv文件的新闻文本，同一件事由于不同的人编写导致差异较大，如何进行去重
12019701659cs commented 2024-12-28 18:20:48 +08:00
Copy Link

可以尝试使用TF-IDF或者或词嵌入对文本进行向量化，然后计算每对新闻文本之间的余弦相似度，找出相似度高于阈值的新闻对，并标记其中一篇为重复。

可以尝试使用TF-IDF或者或词嵌入对文本进行向量化，然后计算每对新闻文本之间的余弦相似度，找出相似度高于阈值的新闻对，并标记其中一篇为重复。
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels   
bug

Something is not working

  
duplicate

This issue or pull request already exists

  
enhancement

New feature

  
help wanted

Need some help

  
invalid

Something is wrong

  
question

More information is needed

  
wontfix

This won't be fixed

No Label
bug
duplicate
enhancement
help wanted
invalid
question
wontfix
Milestone
Clear milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
No Assignees
2 Participants
Notifications
Due Date
The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: HswOAuth/llm_course#448
No description provided.
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?