📄 Llamaguard¶

metrics.llm_as_judge.safety.llamaguard

LLMAsJudge(
    inference_model=CrossProviderInferenceEngine(
        model="llama-3-3-70b-instruct",
        max_tokens=20,
        seed=42,
        temperature=1e-07,
    ),
    template="templates.safety.llamaguard",
    task="rating.single_turn",
    format="formats.chat_api",
    main_score="llama_3_3_70b_instruct_cross_provider_template_llamaguard",
)

[source]

from unitxt.inference import CrossProviderInferenceEngine

Explanation about LLMAsJudge¶

LLM-as-judge-based metric class for evaluating correctness of generated predictions.

This class uses the source prompt given to the generator and the generator’s predictions to evaluate correctness using one of three supported tasks (rating.single_turn, rating.single_turn_with_reference, pairwise_comparative_rating.single_turn).

Attributes:
main_score (str): The main score label used for evaluation.

task (Literal[“rating.single_turn”,”rating.single_turn_with_reference”, “pairwise_comparative_rating.single_turn”]): The type of task the llm as judge runs. This defines the output and input format of the judge model.

template (Template): The template used when generating inputs for the judge llm.

format (Format): The format used when generating inputs for judge llm.

system_prompt (SystemPrompt): The system prompt used when generating inputs for judge llm.

strip_system_prompt_and_format_from_inputs (bool): Whether to strip the system prompt and formatting from the inputs that the models that is being judges received, when they are inserted to the llm-as-judge prompt.

inference_model (InferenceEngine): The module that creates the inference of the judge llm.

reduction_map (dict): A dictionary specifying the reduction method for the metric.

batch_size (int): The size of the bulk.

Explanation about CrossProviderInferenceEngine¶

Inference engine capable of dynamically switching between multiple providers APIs.

This class extends the InferenceEngine and OpenAiInferenceEngineParamsMixin to enable seamless integration with various API providers. The supported APIs are specified in _supported_apis, allowing users to interact with multiple models from different sources. The provider_model_map dictionary maps each API to specific model identifiers, enabling automatic configuration based on user requests.

Current _supported_apis = [“watsonx”, “together-ai”, “open-ai”, “aws”, “ollama”, “bam”, “watsonx-sdk”, “rits”, “vertex-ai”,”hf-local”]

Args:

provider (Optional):
Specifies the current API in use. Must be one of the literals in _supported_apis.

provider_model_map (Dict[_supported_apis, Dict[str, str]]):
mapping each supported API to a corresponding model identifier string. This mapping allows consistent access to models across different API backends.

provider_specific_args:
(Optional[Dict[str, Dict[str,str]]]) Args specific to a provider for example provider_specific_args={“watsonx”: {“max_requests_per_second”: 4}}

References: templates.safety.llamaguard, formats.chat_api