📄 Generic Inference Engine Judges¶

metrics.llm_as_judge.conversation_answer_idk.generic_inference_engine_judges

type: LLMAsJudge
inference_model: 
  type: GenericInferenceEngine
template: templates.response_assessment.judges.idk.v1
task: rating.single_turn
main_score: metrics.llm_as_judge.rating.generic_template_v1
prediction_type: str
[source]

Explanation about LLMAsJudge¶

LLM-as-judge-based metric class for evaluating correctness of generated predictions.

This class uses the source prompt given to the generator and the generator’s predictions to evaluate correctness using one of three supported tasks (rating.single_turn, rating.single_turn_with_reference, pairwise_comparative_rating.single_turn).

Attributes:
main_score (str): The main score label used for evaluation.

task (Literal[“rating.single_turn”,”rating.single_turn_with_reference”, “pairwise_comparative_rating.single_turn”]): The type of task the llm as judge runs. This defines the output and input format of the judge model.

template (Template): The template used when generating inputs for the judge llm.

format (Format): The format used when generating inputs for judge llm.

system_prompt (SystemPrompt): The system prompt used when generating inputs for judge llm.

strip_system_prompt_and_format_from_inputs (bool): Whether to strip the system prompt and formatting from the inputs that the models that is being judges received, when they are inserted to the llm-as-judge prompt.

inference_model (InferenceEngine): The module that creates the inference of the judge llm.

reduction_map (dict): A dictionary specifying the reduction method for the metric.

batch_size (int): The size of the bulk.

References: templates.response_assessment.judges.idk.v1