📄 Llama 3 3 70B Instruct Judge¶

metrics.rag.response_generation.answer_completeness.llama_3_3_70b_instruct_judge

type: LLMJudgeDirect
inference_engine: 
  type: CrossProviderInferenceEngine
  model: llama-3-3-70b-instruct
  max_tokens: 1024
criteria: 
  type: CriteriaWithOptions
  name: answer_completeness
  description: The response is complete: all the aspects of the reference answer are addressed in the response. The response might use different phrasing or wording from the reference answer.
  options: 
    - type: CriteriaOption
      name: Excellent
      description: The response addresses all aspects of the reference answer.
    - type: CriteriaOption
      name: Good
      description: The response addresses most aspects of the reference answer, with minor omissions.
    - type: CriteriaOption
      name: mediocre
      description: The response covers the essential aspects of the reference answer but has notable omissions.
    - type: CriteriaOption
      name: Bad
      description: The response covers only a few aspects of the reference answer, with significant omissions.
    - type: CriteriaOption
      name: Very Bad
      description: The response fails to address the reference answer meaningfully, with most aspects omitted.
  option_map: 
    Excellent: 1.0
    Good: 0.75
    mediocre: 0.5
    Bad: 0.25
    "Very Bad": 0
context_fields: 
  question: question
  reference_answers: reference_answers
criteria_field: criteria
generate_summaries: False
check_positional_bias: False
[source]

Explanation about CrossProviderInferenceEngine¶

Inference engine capable of dynamically switching between multiple providers APIs.

This class extends the InferenceEngine and OpenAiInferenceEngineParamsMixin to enable seamless integration with various API providers. The supported APIs are specified in _supported_apis, allowing users to interact with multiple models from different sources. The provider_model_map dictionary maps each API to specific model identifiers, enabling automatic configuration based on user requests.

Current _supported_apis = [“watsonx”, “together-ai”, “open-ai”, “aws”, “ollama”, “bam”, “watsonx-sdk”, “rits”, “vertex-ai”]

Args:

provider (Optional):
Specifies the current API in use. Must be one of the literals in _supported_apis.

provider_model_map (Dict[_supported_apis, Dict[str, str]]):
mapping each supported API to a corresponding model identifier string. This mapping allows consistent access to models across different API backends.