📄 Relevance¶

metrics.llm_as_judge.direct.criteria.relevance

CriteriaWithOptions(
    name="relevance",
    description="Is the response relevant with respect to the article? The response captures the key points of the article. Consider whether all and only the important aspects are contained in the response. Penalize responses that contain redundancies or excess information.",
    prediction_field="response",
    context_fields=[
        "article",
    ],
    options=[
        CriteriaOption(
            name="1",
            description="The response is not relevant at all to the article.",
        ),
        CriteriaOption(
            name="2",
            description="The response is somewhat relevant to the article.",
        ),
        CriteriaOption(
            name="3",
            description="The response is relevant to the article.",
        ),
        CriteriaOption(
            name="4",
            description="The response is very relevant to the article.",
        ),
        CriteriaOption(
            name="5",
            description="The response is exceptionally relevant to the article and contains only the important aspects.",
        ),
    ],
    option_map={
        "1": 0.0,
        "2": 0.25,
        "3": 0.5,
        "4": 0.75,
        "5": 1.0,
    },
)

[source]

from unitxt.llm_as_judge_constants import CriteriaOption

Explanation about CriteriaWithOptions¶

Criteria used by DirectLLMJudge to run evaluations.

Explanation about CriteriaOption¶

A criteria option.