📄 Llama 3 3 70B Instruct Wml C A Logprobs¶

metrics.rag.faithfulness.llama_3_3_70b_instruct_wml_c_a_logprobs

TaskBasedLLMasJudge(
    inference_model="engines.classification.llama_3_3_70b_instruct_wml",
    template="templates.rag_eval.faithfulness.judge_no_question_simplified_logprobs",
    task="tasks.rag_eval.faithfulness.binary",
    format=None,
    main_score="faithfulness_c_a_logprobs",
    prediction_field="answer",
    infer_log_probs=True,
)
[source]

Explanation about TaskBasedLLMasJudge¶

LLM-as-judge-based metric class for evaluating correctness of generated predictions.

This class can use any task and matching template to evaluate the predictions. All task/templates field are taken from the instance’s task_data. The instances sent to the judge can either be: 1.a unitxt dataset, in which case the predictions are copied to a specified field of the task. 2. dictionaries with the fields required by the task and template.

Args:

main_score (str):
The main score label used for evaluation.

task (str):
The type of task the llm as judge runs. This defines the output and input format of the judge model.

template (Template):
The template used when generating inputs for the judge llm.

format (Format):
The format used when generating inputs for judge llm.

system_prompt (SystemPrompt):
The system prompt used when generating inputs for judge llm.

strip_system_prompt_and_format_from_inputs (bool):
Whether to strip the system prompt and formatting from the inputs that the models that is being judges received, when they are inserted to the llm-as-judge prompt.

inference_model (InferenceEngine):
The module that creates the inference of the judge llm.

reduction_map (dict):
A dictionary specifying the reduction method for the metric.

batch_size (int):
The size of the bulk.

infer_log_probs(bool):
whether to perform the inference using logprobs. If true, the template’s post-processing must support the logprobs output.

judge_to_generator_fields_mapping (Dict[str, str]):
optional mapping between the names of the fields in the generator task and the judge task. For example, if the generator task uses “reference_answers” and the judge task expect “ground_truth”, include {“ground_truth”: “reference_answers”} in this dictionary.

prediction_field (str):
if indicated, and prediction exist, copy prediction to this field name in task_data.

include_meta_data (bool):
whether to include the inference per-instance metadata in the returned results.

References: templates.rag_eval.faithfulness.judge_no_question_simplified_logprobs, engines.classification.llama_3_3_70b_instruct_wml, tasks.rag_eval.faithfulness.binary