📄 Coherence¶

metrics.llm_as_judge.direct.criteria.coherence

CriteriaWithOptions(
    name="coherence",
    description="Is the response coherent with respect to the original text?",
    prediction_field="response",
    context_fields=[
        "original text",
    ],
    options=[
        CriteriaOption(
            name="1",
            description="The response lacks coherence and detail, failing to accurately capture the main points of the original text. It may contain grammatical errors or inaccuracies.",
        ),
        CriteriaOption(
            name="2",
            description="The response provides a slightly improved restatement of the original text compared to score 1 but still lacks coherence and may contain inaccuracies or omissions.",
        ),
        CriteriaOption(
            name="3",
            description="The response captures the main points of the original text with moderate accuracy and coherence, offering a clearer understanding of the central events and relationships depicted.",
        ),
        CriteriaOption(
            name="4",
            description="The response effectively conveys the main points of the original text with good accuracy and coherence, providing a clear overview of the events and relationships.",
        ),
        CriteriaOption(
            name="5",
            description="The response demonstrates a high level of accuracy and coherence, effectively conveying the main points of the original text in a concise and clear manner.",
        ),
    ],
    option_map={
        "1": 0.0,
        "2": 0.25,
        "3": 0.5,
        "4": 0.75,
        "5": 1,
    },
)

[source]

from unitxt.llm_as_judge_constants import CriteriaOption

Explanation about CriteriaWithOptions¶

Criteria used by DirectLLMJudge to run evaluations.

Explanation about CriteriaOption¶

A criteria option.