unitxt.metric_utils module

class unitxt.metric_utils.FromPredictionsAndOriginalData(__tags__: Dict[str, str] = {}, caching: bool = None)

Bases: StreamInitializerOperator

class unitxt.metric_utils.InstanceInput(references: List[Any], additional_inputs: Dict | None = None)

Bases: Dataclass

A single instance inputted to a metric service.

class unitxt.metric_utils.MetricRecipe(__tags__: ~typing.Dict[str, str] = {}, caching: bool = None, steps: ~typing.List[~unitxt.operator.StreamingOperator], calc_confidence_intervals: bool = True)

Bases: SequentialOperatorInitializer

class unitxt.metric_utils.MetricRequest

Bases: Dataclass

A request to a metrics service, includes a list of input instances.

class unitxt.metric_utils.MetricResponse(global_score: Dict[str, Any])

Bases: Dataclass

A response produced by a metrics service, includes the computed scores.

class unitxt.metric_utils.MultiStreamScoreMean(__tags__: Dict[str, str] = {}, caching: bool = None)

Bases: MultiStreamOperator

unitxt.metric_utils.get_remote_metrics_endpoint() → str

Load the remote metrics endpoint from an environment variable.

Returns:

str - The remote endpoint on which the remote metrics are available.

unitxt.metric_utils.get_remote_metrics_names() → List[str]

Load the remote metrics names from an environment variable.

Returns:

List[str] - names of metrics to be executed remotely.