📄 Llama3 V1 Ibmgenai Judges¶

metrics.llm_as_judge.conversation_answer_idk.llama3_v1_ibmgenai_judges

LLMAsJudge(
    inference_model=IbmGenAiInferenceEngine(
        model_name="meta-llama/llama-3-70b-instruct",
        parameters=IbmGenAiInferenceEngineParams(
            max_new_tokens=256,
        ),
    ),
    template="templates.response_assessment.judges.idk.v1",
    task="rating.single_turn",
    main_score="metrics.llm_as_judge.rating.llama370binstruct_template_v1",
    prediction_type="str",
)

[source]

from unitxt.inference import IbmGenAiInferenceEngine, IbmGenAiInferenceEngineParams

Explanation about LLMAsJudge¶

LLM-as-judge-based metric class for evaluating correctness of generated predictions.

This class uses the source prompt given to the generator and the generator’s predictions to evaluate correctness using one of three supported tasks (rating.single_turn, rating.single_turn_with_reference, pairwise_comparative_rating.single_turn).

Attributes:
main_score (str): The main score label used for evaluation.

task (Literal[“rating.single_turn”,”rating.single_turn_with_reference”, “pairwise_comparative_rating.single_turn”]): The type of task the llm as judge runs. This defines the output and input format of the judge model.

template (Template): The template used when generating inputs for the judge llm.

format (Format): The format used when generating inputs for judge llm.

system_prompt (SystemPrompt): The system prompt used when generating inputs for judge llm.

strip_system_prompt_and_format_from_inputs (bool): Whether to strip the system prompt and formatting from the inputs that the models that is being judges received, when they are inserted to the llm-as-judge prompt.

inference_model (InferenceEngine): The module that creates the inference of the judge llm.

reduction_map (dict): A dictionary specifying the reduction method for the metric.

batch_size (int): The size of the bulk.

References: templates.response_assessment.judges.idk.v1