【求助贴】使用ragas评估测试的时候报错，求帮助解决 #251

New Issue

11256832216cs · 2024-10-21T16:16:24+08:00

11256832216cs commented

2024-10-21 16:16:24 +08:00

代码如下：

from dotenv import load_dotenv
load_dotenv()

from datasets import Dataset

data = {
"question": ["What is the capital of France?"],
"answer": ["The capital of France is Paris."],
"retrieved_contexts": [["Paris is the capital city of France."]],
"ground_truth": ["Paris is the capital city of France."]
}
print(data)
dataset = Dataset.from_dict(data)
print(dataset)
print("-------------------------------------------")

# 进行评估
from ragas import evaluate
from ragas.metrics import (
    faithfulness,
    answer_relevancy,
    context_recall,
    context_precision,
)

result = evaluate(
    dataset,
    metrics=[
        context_precision,
        context_recall,
        faithfulness,
        answer_relevancy,
    ],
)

print(result)
df = result.to_pandas()
df.head()

运行后报错如下：


Evaluating:   0%|          | 0/4 [00:00<?, ?it/s]Exception raised in Job[0]: ValidationError(1 validation error for ChatMessage
role
  none is not an allowed value (type=type_error.none.not_allowed))
Evaluating:  25%|██▌       | 1/4 [00:02<00:08,  2.97s/it]Exception raised in Job[3]: ValidationError(1 validation error for ChatMessage
role
  none is not an allowed value (type=type_error.none.not_allowed))
Evaluating:  50%|█████     | 2/4 [00:03<00:03,  1.64s/it]Exception raised in Job[2]: ValidationError(1 validation error for ChatMessage
role
  none is not an allowed value (type=type_error.none.not_allowed))
Evaluating:  75%|███████▌  | 3/4 [00:04<00:01,  1.18s/it]Exception raised in Job[1]: ValidationError(1 validation error for ChatMessage
role
  none is not an allowed value (type=type_error.none.not_allowed))
Evaluating: 100%|██████████| 4/4 [00:04<00:00,  1.16s/it]
{'context_precision': nan, 'context_recall': nan, 'faithfulness': nan, 'answer_relevancy': nan}

该报错直接导致评估失败，所有指标输出为nan，但在ragas的项目文档中没有找到role相关的设置

### 代码如下： ``` from dotenv import load_dotenv load_dotenv() from datasets import Dataset data = { "question": ["What is the capital of France?"], "answer": ["The capital of France is Paris."], "retrieved_contexts": [["Paris is the capital city of France."]], "ground_truth": ["Paris is the capital city of France."] } print(data) dataset = Dataset.from_dict(data) print(dataset) print("-------------------------------------------") # 进行评估 from ragas import evaluate from ragas.metrics import ( faithfulness, answer_relevancy, context_recall, context_precision, ) result = evaluate( dataset, metrics=[ context_precision, context_recall, faithfulness, answer_relevancy, ], ) print(result) df = result.to_pandas() df.head() ``` ### 运行后报错如下： ``` Evaluating: 0%| | 0/4 [00:00<?, ?it/s]Exception raised in Job[0]: ValidationError(1 validation error for ChatMessage role none is not an allowed value (type=type_error.none.not_allowed)) Evaluating: 25%|██▌ | 1/4 [00:02<00:08, 2.97s/it]Exception raised in Job[3]: ValidationError(1 validation error for ChatMessage role none is not an allowed value (type=type_error.none.not_allowed)) Evaluating: 50%|█████ | 2/4 [00:03<00:03, 1.64s/it]Exception raised in Job[2]: ValidationError(1 validation error for ChatMessage role none is not an allowed value (type=type_error.none.not_allowed)) Evaluating: 75%|███████▌ | 3/4 [00:04<00:01, 1.18s/it]Exception raised in Job[1]: ValidationError(1 validation error for ChatMessage role none is not an allowed value (type=type_error.none.not_allowed)) Evaluating: 100%|██████████| 4/4 [00:04<00:00, 1.16s/it] {'context_precision': nan, 'context_recall': nan, 'faithfulness': nan, 'answer_relevancy': nan} ``` 该报错直接导致评估失败，所有指标输出为nan，但在ragas的项目文档中没有找到role相关的设置

binah commented

2024-10-22 11:27:03 +08:00

这边构建了程序之后运行没能得到预期的错误，推测可能是版本问题。特别是pydantic和ragas

11256832216cs closed this issue

2024-10-22 18:06:29 +08:00

Sign in to join this conversation.