📄 Step By Step Reasoning Commonsense¶

metrics.llm_as_judge.direct.criteria.step_by_step_reasoning_commonsense

CriteriaWithOptions(
    name="step_by_step_reasoning_commonsense",
    description="Does this step contain any errors in relation to general knowledge about the world (i.e. how to compute velocity, how many inches in one foot, etc) not explicitly provided in the context?",
    prediction_field="step",
    context_fields=[
        "question",
        "premise",
        "hypothesis",
        "model reasoning",
        "correct answer",
    ],
    options=[
        CriteriaOption(
            name="Yes",
            description="",
        ),
        CriteriaOption(
            name="No",
            description="",
        ),
    ],
    option_map={
        "Yes": 0.0,
        "No": 1.0,
    },
)

[source]

from unitxt.llm_as_judge_constants import CriteriaOption

Explanation about CriteriaWithOptions¶

Criteria used by DirectLLMJudge to run evaluations.

Explanation about CriteriaOption¶

A criteria option.