📄 Translation Quality¶

metrics.llm_as_judge.direct.criteria.translation_quality

CriteriaWithOptions(
    name="translation_quality",
    description="On a scale from 0 to 6, is the translation of the source text accurate, fluent, comprenhencible and free of errors?
Accuracy: How well does the translation convey the original meaning and content of the source text?
Fluency: How natural and idiomatic is the translation in terms of grammar, syntax, and phrasing?
Comprehensibility: How easily can the translation be understood by a native speaker of the target language?
Errors: Are there any errors in grammar, vocabulary, punctuation, or formatting that affect the overall quality of the translation?",
    prediction_field="translation",
    context_fields=[
        "source language",
        "target language",
        "source text",
        "reference translation",
    ],
    options=[
        CriteriaOption(
            name="0",
            description="Nonsense/No meaning preserved: Nearly all information is lost between the translation and the source text. Grammar is irrelevant.",
        ),
        CriteriaOption(
            name="1",
            description="Minimal Meaning Preserved: Only isolated fragments of meaning are retained. The translation is largely incoherent and fails to convey the main ideas. Grammar is poor or broken.",
        ),
        CriteriaOption(
            name="2",
            description="Some Meaning Preserved: The translation preserves some of the meaning of the source text but misses significant parts. The narrative is hard to follow due to fundamental errors. Grammar may be poor.",
        ),
        CriteriaOption(
            name="3",
            description="Moderate Meaning Preserved: The core message is partially conveyed, but there are frequent issues with grammar, fluency, or comprehension that impact the overall readability and accuracy.",
        ),
        CriteriaOption(
            name="4",
            description="Most Meaning Preserved and Few Grammar Mistakes: The translation retains most of the meaning of the source text. It may have some grammar mistakes or minor contextual inconsistencies.",
        ),
        CriteriaOption(
            name="5",
            description="Nearly Perfect: The translation is highly accurate and mostly fluent. Only very minor grammar or phrasing issues are present, and they do not hinder understanding.",
        ),
        CriteriaOption(
            name="6",
            description="Perfect Meaning and Grammar: The meaning of the translation is completely consistent with the source text and the surrounding context (if applicable). The grammar is also correct.",
        ),
    ],
    option_map={
        "0": 0.0,
        "1": 0.16666666666666666,
        "2": 0.3333333333333333,
        "3": 0.5,
        "4": 0.6666666666666666,
        "5": 0.8333333333333334,
        "6": 1.0,
    },
)

[source]

from unitxt.llm_as_judge_constants import CriteriaOption

Explanation about CriteriaWithOptions¶

Criteria used by DirectLLMJudge to run evaluations.

Explanation about CriteriaOption¶

A criteria option.