📄 Llama 3 3 70B Instruct Judge¶

metrics.rag.response_generation.answer_completeness.llama_3_3_70b_instruct_judge

LLMJudgeDirect(
    inference_engine=CrossProviderInferenceEngine(
        model="llama-3-3-70b-instruct",
        max_tokens=1024,
        temperature=0,
    ),
    criteria=CriteriaWithOptions(
        name="answer_completeness",
        description="The response is complete: all the aspects of the reference answer are addressed in the response. The response might use different phrasing or wording from the reference answer.",
        prediction_field=None,
        context_fields=None,
        options=[
            CriteriaOption(
                name="Excellent",
                description="The response addresses all aspects of the reference answer.",
            ),
            CriteriaOption(
                name="Good",
                description="The response addresses most aspects of the reference answer, with minor omissions.",
            ),
            CriteriaOption(
                name="mediocre",
                description="The response covers the essential aspects of the reference answer but has notable omissions.",
            ),
            CriteriaOption(
                name="Bad",
                description="The response covers only a few aspects of the reference answer, with significant omissions.",
            ),
            CriteriaOption(
                name="Very Bad",
                description="The response fails to address the reference answer meaningfully, with most aspects omitted.",
            ),
        ],
        option_map={
            "Excellent": 1.0,
            "Good": 0.75,
            "mediocre": 0.5,
            "Bad": 0.25,
            "Very Bad": 0,
        },
    ),
    context_fields={
        "question": "question",
        "reference_answers": "reference_answers",
    },
    criteria_field="criteria",
    generate_summaries=False,
    check_positional_bias=False,
)

[source]

from unitxt.inference import CrossProviderInferenceEngine
from unitxt.llm_as_judge_constants import CriteriaOption, CriteriaWithOptions

Explanation about LLMJudgeDirect¶

LLMJudgeDirect is a specialized evaluation metric that performs Direct Assessment using an LLM to score responses based on a predefined evaluation criteria.

Direct Assessment is an evaluation paradigm in which the LLM selects one of a predefined set of options based on an assessment criterion. This approach can be used for Likert-scale scoring (e.g., 1-5) or selecting from semantically conditioned literals (e.g., Yes/No, Pass/Fail).

Explanation about CrossProviderInferenceEngine¶

Inference engine capable of dynamically switching between multiple providers APIs.

This class extends the InferenceEngine and OpenAiInferenceEngineParamsMixin to enable seamless integration with various API providers. The supported APIs are specified in _supported_apis, allowing users to interact with multiple models from different sources. The provider_model_map dictionary maps each API to specific model identifiers, enabling automatic configuration based on user requests.

Current _supported_apis = [“watsonx”, “together-ai”, “open-ai”, “aws”, “ollama”, “bam”, “watsonx-sdk”, “rits”, “vertex-ai”,”hf-local”]

Args:

provider (Optional):
Specifies the current API in use. Must be one of the literals in _supported_apis.

provider_model_map (Dict[_supported_apis, Dict[str, str]]):
mapping each supported API to a corresponding model identifier string. This mapping allows consistent access to models across different API backends.

provider_specific_args:
(Optional[Dict[str, Dict[str,str]]]) Args specific to a provider for example provider_specific_args={“watsonx”: {“max_requests_per_second”: 4}}

Explanation about CriteriaWithOptions¶

Criteria used by DirectLLMJudge to run evaluations.

Explanation about CriteriaOption¶

A criteria option.