unitxt.metric_utils module

class unitxt.metric_utils.DeleteTargetPrefix(data_classification_policy: List[str] = None, _requirements_list: List[str] | Dict[str, str] = [], caching: bool = None, apply_to_streams: List[str] = None, dont_apply_to_streams: List[str] = None)

Bases: InstanceOperator, ArtifactFetcherMixin

class unitxt.metric_utils.FromPredictionsAndOriginalData(data_classification_policy: List[str] = None, _requirements_list: List[str] | Dict[str, str] = [], caching: bool = None)

Bases: StreamInitializerOperator

class unitxt.metric_utils.InstanceInput(prediction: Any = __required__, references: List[Any] = __required__, additional_inputs: Dict | NoneType = None)

Bases: Dataclass

A single instance inputted to a metric service.

class unitxt.metric_utils.JoinSubsetsAndGroups(data_classification_policy: List[str] = None, _requirements_list: List[str] | Dict[str, str] = [], caching: bool = None)

Bases: MultiStreamOperator

class unitxt.metric_utils.MetricRecipe(data_classification_policy: List[str] = None, max_steps: int | NoneType = None, steps: List[unitxt.operator.StreamingOperator] = [], _requirements_list: List[str] | Dict[str, str] = [], caching: bool = None, calc_confidence_intervals: bool = True, subset_depth: int = 2)

Bases: SequentialOperatorInitializer

class unitxt.metric_utils.MetricRequest(instance_inputs: List[unitxt.metric_utils.InstanceInput] = __required__)

Bases: Dataclass

A request to a metrics service, includes a list of input instances.

class unitxt.metric_utils.MetricResponse(instances_scores: List[Dict[str, Any]] = __required__, global_score: Dict[str, Any] = __required__)

Bases: Dataclass

A response produced by a metrics service, includes the computed scores.

class unitxt.metric_utils.PostProcessRecipe(data_classification_policy: List[str] = None, max_steps: int | NoneType = None, steps: List[unitxt.operator.StreamingOperator] = [], _requirements_list: List[str] | Dict[str, str] = [], caching: bool = None)

Bases: SequentialOperatorInitializer

class unitxt.metric_utils.SplitSubsetsAndGroups(data_classification_policy: List[str] = None, _requirements_list: List[str] | Dict[str, str] = [], caching: bool = None, subsets_field: str = 'subset', groups_field: str = 'groups', subset_depth: int | NoneType = None)

Bases: MultiStreamOperator

Splits a MultiStream that is small - for metrics, hence: whole stream can sit in memory, split by the value of field ‘group’.

Parameters:

number_of_fusion_generations – int

the value in field group is of the form “sourcen/sourcenminus1/…” describing the sources in which the instance sat when these were fused, potentially several phases of fusion. the name of the most recent source sits first in this value. (See BaseFusion and its extensions) subsets_depth specifies the depth of the prefix by which to split the stream.

unitxt.metric_utils.get_remote_metrics_endpoint() → str

Load the remote metrics endpoint from an environment variable.

Returns:

str - The remote endpoint on which the remote metrics are available.

unitxt.metric_utils.get_remote_metrics_names() → List[str]

Load the remote metrics names from an environment variable.

Returns:

List[str] - names of metrics to be executed remotely.

unitxt.metric_utils.group_str(json_str)

unitxt.metric_utils.group_str_to_key_value(group_str)

unitxt.metric_utils.nan_mean(scores)

unitxt.metric_utils.stream_name_to_origin_subset_group(stream_name)