📄 Deepseek V3¶

metrics.llm_as_judge.direct.rits.deepseek_v3

LLMJudgeDirect(
    inference_engine=CrossProviderInferenceEngine(
        max_tokens=1024,
        seed=42,
        temperature=0,
        provider="rits",
        model="deepseek-v3",
    ),
    evaluator_name="DEEPSEEK_V3",
    generate_summaries=False,
)

[source]

from unitxt.inference import CrossProviderInferenceEngine

Explanation about LLMJudgeDirect¶

LLMJudgeDirect is a specialized evaluation metric that performs Direct Assessment using an LLM to score responses based on a predefined evaluation criteria.

Direct Assessment is an evaluation paradigm in which the LLM selects one of a predefined set of options based on an assessment criterion. This approach can be used for Likert-scale scoring (e.g., 1-5) or selecting from semantically conditioned literals (e.g., Yes/No, Pass/Fail).

Explanation about CrossProviderInferenceEngine¶

Inference engine capable of dynamically switching between multiple providers APIs.

This class extends the InferenceEngine and OpenAiInferenceEngineParamsMixin to enable seamless integration with various API providers. The supported APIs are specified in _supported_apis, allowing users to interact with multiple models from different sources. The provider_model_map dictionary maps each API to specific model identifiers, enabling automatic configuration based on user requests.

Current _supported_apis = [“watsonx”, “together-ai”, “open-ai”, “aws”, “ollama”, “bam”, “watsonx-sdk”, “rits”, “vertex-ai”,”hf-local”]

Args:

provider (Optional):
Specifies the current API in use. Must be one of the literals in _supported_apis.

provider_model_map (Dict[_supported_apis, Dict[str, str]]):
mapping each supported API to a corresponding model identifier string. This mapping allows consistent access to models across different API backends.

provider_specific_args:
(Optional[Dict[str, Dict[str,str]]]) Args specific to a provider for example provider_specific_args={“watsonx”: {“max_requests_per_second”: 4}}