📄 Step By Step Reasoning Bad Final Answer¶

metrics.llm_as_judge.direct.criteria.step_by_step_reasoning_bad_final_answer

CriteriaWithOptions(
    name="step_by_step_reasoning_bad_final_answer",
    description="Does this step contain a final step with an incorrect final answer? (If an explicit 'yes/no' is not provided, an exact match of the correct answer with respect to the question in the context must be given.)",
    prediction_field="step",
    context_fields=[
        "question",
        "premise",
        "hypothesis",
        "model reasoning",
        "correct answer",
    ],
    options=[
        CriteriaOption(
            name="Yes",
            description="",
        ),
        CriteriaOption(
            name="No",
            description="",
        ),
    ],
    option_map={
        "Yes": 0.0,
        "No": 1.0,
    },
)

[source]

from unitxt.llm_as_judge_constants import CriteriaOption

Explanation about CriteriaWithOptions¶

Criteria used by DirectLLMJudge to run evaluations.

Explanation about CriteriaOption¶

A criteria option.