unitxt¶
- unitxt package
- Subpackages
- Submodules
- unitxt.api module
- unitxt.artifact module
AbstractCatalogArtifactArtifactLinkArtifactListCatalogsMissingArtifactTypeErrorUnitxtArtifactNotFoundErrorUnrecognizedArtifactTypeErrordict_diff_string()fetch_artifact()get_artifacts_data_classification()get_catalog_name_and_args()get_closest_artifact_type()get_raw()is_name_legal_for_catalog()maybe_recover_artifact()maybe_recover_artifacts_structure()register_all_artifacts()reset_artifacts_json_cache()verbosed_fetch_artifact()verify_legal_catalog_name()
- unitxt.augmentors module
- unitxt.base_metric module
- unitxt.benchmark module
- unitxt.blocks module
- unitxt.card module
- unitxt.catalog module
- unitxt.collections module
- unitxt.collections_operators module
- unitxt.dataclass module
AbstractFieldAbstractFieldErrorDataclassDataclassMetaFieldFinalFieldFinalFieldErrorInternalFieldMissingDefaultErrorNonPositionalFieldOptionalFieldRequiredFieldRequiredFieldErrorTypeMismatchErrorUndefinedUnexpectedArgumentErrorabstract_fields()asdict()class_fields()external_fields_names()fields()fields_names()final_fields()get_field_default()get_fields()is_abstract_field()is_class_method()is_dataclass()is_final_field()is_possible_field()required_fields()to_dict()
- unitxt.dataset module
- unitxt.dataset_utils module
- unitxt.deprecation_utils module
- unitxt.dialog_operators module
- unitxt.dict_utils module
- unitxt.error_utils module
- unitxt.eval_utils module
- unitxt.evaluate_cli module
cli_load_dataset()configure_unitxt_settings()extract_overwrite_args()extract_scores()initialize_inference_engine()main()prepare_kwargs()prepare_output_paths()prepend_timestamp_to_path()process_and_save_results()run_evaluation()run_inference()setup_logging()setup_parser()summarize_cli()try_parse_json()
- unitxt.file_utils module
- unitxt.formats module
- unitxt.fusion module
- unitxt.generator_utils module
- unitxt.hf_utils module
- unitxt.image_operators module
- unitxt.inference module
AsyncTokenBucketAzureOpenAIInferenceEngineCredentialsOpenAiCrossProviderInferenceEngineGenericInferenceEngineHFAutoModelInferenceEngineHFGenerationParamsMixinHFInferenceEngineBaseHFLlavaInferenceEngineHFOptionSelectingInferenceEngineHFPeftInferenceEngineHFPipelineBasedInferenceEngineInferenceEngineLMMSEvalBaseInferenceEngineLMMSEvalInferenceEngineLMMSEvalLoglikelihoodInferenceEngineLazyLoadMixinListWithMetadataLiteLLMInferenceEngineLogProbInferenceEngineMetricInferenceEngineMockInferenceEngineMockModeMixinOllamaInferenceEngineOpenAiInferenceEngineOpenAiInferenceEngineParamsMixinOptionSelectingByLogProbsInferenceEngineRITSInferenceEngineStandardAPIParamsMixinTextGenerationInferenceOutputTogetherAiInferenceEngineTogetherAiInferenceEngineParamsMixinTorchDeviceMixinVLLMInferenceEngineVLLMParamsMixinVLLMRemoteInferenceEngineWMLChatParamsMixinWMLGenerationParamsMixinWMLInferenceEngineBaseWMLInferenceEngineChatWMLInferenceEngineGenerationbatched()get_images_without_text()get_model_and_label_id()get_text_without_images()mock_logprobs_default_value_factory()run_with_imap()
- unitxt.instructions module
- unitxt.llm_as_judge module
- unitxt.llm_as_judge_chat_templates module
- unitxt.llm_as_judge_constants module
- unitxt.llm_as_judge_from_template module
- unitxt.llm_as_judge_operators module
- unitxt.llm_as_judge_utils module
- unitxt.loaders module
- Loaders: Generators of Unitxt Multistreams from existing date sources
LazyLoaderLoadCSVLoadFromAPILoadFromDictionaryLoadFromHFSpaceLoadFromIBMCloudLoadFromKaggleLoadFromSklearnLoadHFLoadIOBLoadJsonFileLoadWithPandasLoaderMissingKaggleCredentialsErrorMultipleSourceLoaderTURLColumnTypeAnnotationLoaderUnitxtUnverifiedCodeErrorhf_get_dataset_splits()hf_load_dataset()read_file()
- unitxt.logging_utils module
- unitxt.metric module
- unitxt.metric_utils module
DeleteTargetPrefixEmptyPredictionEvaluationResultsFromPredictionsAndOriginalDataGlobalScoresGroupsScoresInstanceInputInstanceScoresJoinSubsetsAndGroupsMetricRecipeMetricRequestMetricResponsePostProcessRecipeSplitSubsetsAndGroupsSubsetsScoresempty_predictions_generator()get_remote_metrics_endpoint()get_remote_metrics_names()group_str()group_str_to_key_value()nan_mean()stream_name_to_origin_subset_group()
- unitxt.metrics module
ANLSAccuracyAccuracyFastAggregationReductionBertScoreBinaryAccuracyBinaryMaxAccuracyBinaryMaxF1BulkInstanceMetricCharEditDistanceCharEditDistanceAccuracyConfidenceIntervalMixinCorrelationMetricCustomF1CustomF1FuzzyDetectorDictReductionEvaluationInputExactMatchMMF1F1BinaryF1BinaryPosOnlyF1FastF1MacroF1MacroMultiLabelF1MicroF1MicroMultiLabelF1MultiLabelF1StringsF1WeightedFaithfulnessHHEMFinQAEvalFixedGroupAbsvalNormCohensHParaphraseAccuracyFixedGroupAbsvalNormCohensHParaphraseStringContainmentFixedGroupAbsvalNormHedgesGParaphraseAccuracyFixedGroupAbsvalNormHedgesGParaphraseStringContainmentFixedGroupMeanAccuracyFixedGroupMeanBaselineAccuracyFixedGroupMeanBaselineStringContainmentFixedGroupMeanParaphraseAccuracyFixedGroupMeanParaphraseStringContainmentFixedGroupMeanStringContainmentFixedGroupNormCohensHParaphraseAccuracyFixedGroupNormCohensHParaphraseStringContainmentFixedGroupNormHedgesGParaphraseAccuracyFixedGroupNormHedgesGParaphraseStringContainmentFixedGroupPDRParaphraseAccuracyFixedGroupPDRParaphraseStringContainmentFuzzyNerGlobalMetricGraniteGuardianAgenticRiskGraniteGuardianAssistantRiskGraniteGuardianBaseGraniteGuardianCustomRiskGraniteGuardianRagRiskGraniteGuardianUserRiskGroupMeanGroupMeanAccuracyGroupMeanStringContainmentGroupMeanTokenOverlapGroupMetricGroupReductionHuggingfaceBulkMetricHuggingfaceInstanceMetricHuggingfaceMetricInstanceMetricIsCodeMixedJaccardIndexJaccardIndexStringKPAKendallTauMetricKeyValueExtractionLlamaIndexCorrectnessLlamaIndexFaithfulnessLlamaIndexLLMMetricMAPMRRMapReduceMetricMatthewsCorrelationMaxAccuracyMaxReductionMeanReductionMeanSquaredErrorMeteorMeteorFastMetricBasedNerMetricPipelineMetricWithConfidenceIntervalMetricsEnsembleMetricsListMultiTurnMetricMultiTurnToolCallingMetricNDCGNERNLTKMixinNormalizedSacrebleuPearsonrPerplexityPrecisionBinaryPrecisionMacroMultiLabelPrecisionMicroMultiLabelPredictionLengthRandomForestMetricsEnsembleRecallBinaryRecallMacroMultiLabelRecallMicroMultiLabelReductionInstanceMetricReflectionToolCallingMetricReflectionToolCallingMetricSyntacticReflectionToolCallingMixinRegardMetricRelaxedCorrectnessRemoteMetricRerankRecallRetrievalAtKRetrievalMetricRewardRiskTypeRocAucRootMeanReductionRootMeanSquaredErrorRougeRougeHFSQLExecutionAccuracySQLExecutionLogicAccuracySQLNonExecutionAccuracySacreBleuStatsSafetyMetricSentenceBertSequentialSuccessSpearmanrSquadStatisticStringContainmentStringContainmentOldStringContainmentRatioTokenOverlapToolCallKeyValueExtractionToolCallingMetricUnsortedListExactMatchUpdateStreamWebsrcSquadF1WeightedWinRateCorrelationWerabstract_factory()abstract_field()get_index_or_default()hf_evaluate_load()interpret_effect_size()is_original_key()mean_subgroup_score()nan_max()nan_mean()nan_std()new_random_generator()normalize_answer()normalized_cohens_h()normalized_hedges_g()parse_string_types_instead_of_actual_objects()performance_drop_rate()pytrec_eval_at_k()validate_subgroup_types()
- unitxt.normalizers module
- unitxt.operator module
InstanceOperatorInstanceOperatorValidatorInstanceOperatorWithMultiStreamAccessMissingRequirementsErrorMultiStreamOperatorOperatorOperatorErrorPackageRequirementsMixinPagedStreamOperatorSequentialMixinSequentialOperatorSequentialOperatorInitializerSideEffectOperatorSingleStreamOperatorSingleStreamReducerSourceOperatorSourceSequentialOperatorStreamInitializerOperatorStreamOperatorStreamingOperatorinstance_generator()instance_result()stream_single()
- unitxt.operators module
- Operators: Building Blocks of Unitxt Processing Pipelines
AddConstantAddIDAddIncrementalIdApplyApplyMetricApplyOperatorsFieldApplyStreamOperatorsFieldArtifactFetcherMixinBalanceBytesToStringCastCastFieldsCollateInstancesCollateInstancesByFieldComputeExpressionMixinCopyDeduplicateDefaultPlaceHolderDeterministicBalancerDivideAllFieldsByDownloadErrorDownloadOperatorDuplicateInstancesEncodeLabelsExecuteExpressionExtractFieldValuesExtractMostCommonFieldValuesExtractZipFileFeatureGroupedShuffleFieldOperatorFillnaFilterByConditionFilterByConditionBasedOnFieldsFilterByExpressionFixJsonSchemaOfParameterTypesFlattenInstancesFromIterablesGetItemByIndexIndexOfInstanceFieldOperatorInterleaveListsToDialogOperatorIntersectIntersectCorrespondingFieldsIterableSourceJoinStrLengthBalancerListFieldValuesMapInstanceValuesMapValuesMergeStreamsMinimumOneExamplePerLabelRefinerMovePerturbReadFileRecursiveCopyRecursiveReplaceRemoveFieldsRemoveValuesRenameSelectFieldsSetShuffleShuffleFieldValuesSplitByNestedGroupStreamRefinerTakeByFieldUnexpectedHttpCodeErrorWikipediaFetcherZipFieldValuesrecursive_key_value_replace()update_scores_of_stream_instances()
- unitxt.parsing_utils module
- unitxt.processors module
AddPrefixCapitalizeConvertToBooleanDictOfListsToPairsExtractArenaHardNumericalJudgmentExtractHarmRatingJudgementExtractMtBenchLabelJudgmentExtractMtBenchRatingJudgmentExtractSafeUnsafeJudgmentExtractVerbalJudgementBadGoodExtractVerbalJudgmentExtractWithRegexFirstCharacterFixWhiteSpaceGetSQLGetStringAfterGroupDictWithRegexInferDictsToBinaryLogprobsListToEmptyEntitiesTuplesLiteralEvalLowerLowerCaseTillPuncMatchClosestOptionPostProcessRegexParserRemoveArticlesRemovePunctuationsScaleNumberToZeroOneReturnZeroIfFailsSplitStripStanceToProConStrToFloatFormatStringEqualsSubstringTakeFirstNonEmptyLineTakeFirstWordTakeLastNonEmptyLineToListByCommaToListByCommaSpaceToStringToStringStrippedToYesOrNoneUpperYesNoToIntYesToOneElseZeroprocess_instance_value()
- unitxt.random_utils module
- unitxt.recipe module
- unitxt.register module
- unitxt.schema module
- unitxt.serializers module
ConversationSerializerDefaultListSerializerDefaultSerializerDialogSerializerDictAsJsonSerializerDocumentSerializerImageSerializerListSerializerMultiDocumentSerializerMultiTypeSerializerNumberQuantizingSerializerNumberSerializerSQLDatabaseAsSchemaSerializerSerializerSingleTypeSerializerTableSerializerToolCallSerializerToolsSerializerVideoSerializer
- unitxt.settings_utils module
- unitxt.span_lableing_operators module
- unitxt.split_utils module
- unitxt.splitters module
- unitxt.standard module
- unitxt.stream module
- unitxt.stream_operators module
- unitxt.string_operators module
- unitxt.struct_data_operators module
ConstructTableFromRowsColsConvertTableColNamesToSequentialDumpJsonDuplicateTableColumnsDuplicateTableRowsGetNumOfTableCellsInsertEmptyTableRowsJsonStrToDictListToKeyValPairsLoadJsonMapHTMLTableToJSONMapTableListsToStdTableJSONMaskColumnsNamesMultipleToolCallPostProcessorParseCSVPythonCallProcessorSerializeKeyValPairsSerializeTableSerializeTableAsConcatenationSerializeTableAsDFLoaderSerializeTableAsHTMLSerializeTableAsImageSerializeTableAsIndexedRowMajorSerializeTableAsJsonSerializeTableAsMarkdownSerializeTableRowAsListSerializeTableRowAsTextSerializeTriplesShuffleColumnsNamesShuffleTableColumnsShuffleTableRowsToolCallPostProcessorTransposeTableTruncateTableCellsTruncateTableRowsextract_possible_json_str()shuffle_columns()shuffle_rows()truncate_cell()
- unitxt.system_prompts module
- unitxt.task module
- unitxt.templates module
ApplyRandomTemplateApplySingleTemplateApplyTemplateDialogFieldsDataDialogPairwiseChoiceTemplateDialogTemplateInputFormatTemplateInputOutputTemplateInputOutputTemplateWithCustomTargetJsonOutputFormatTemplateJsonOutputTemplateKeyValTemplateMultiLabelTemplateMultiReferenceTemplateMultiTurnTemplateMultipleChoiceTemplateNullTemplateOutputFormatTemplateOutputQuantizingTemplatePairwiseChoiceTemplatePairwiseComparativeRatingTemplateSpanLabelingBaseTemplateSpanLabelingJsonTemplateSpanLabelingTemplateTemplateTemplateFormatKeyErrorTemplatesDictTemplatesListYesNoTemplateescape_chars()random()
- unitxt.text2sql_utils module
CacheDatabaseConnectorInMemoryDatabaseConnectorLocalSQLiteConnectorRemoteDatabaseConnectorSQLExecutionResultSQLNonExecutionMetricResultcollect_clause()compare_dfs_bird_eval_logic()compare_dfs_ignore_colnames_ordered_rows()compare_dfs_ignore_colnames_subset()compare_dfs_ignore_colnames_unordered_rows()compare_result_dfs()execute_query_local()execute_query_remote()extract_select_columns()extract_select_info()extract_sql_from_text()generate_cache_key()get_cache()get_db_connector()get_sql_execution_results()is_sqlglot_parsable()is_sqlparse_parsable()replace_select_clause()run_query()sql_exact_match()sqlglot_optimized_equivalence()sqlglot_parsed_queries_equivalent()sqlparse_queries_equivalent()strip_alias()
- unitxt.text_utils module
- unitxt.type_utils module
GenericTypedDictNormalizedTypeUnsupportedTypeErrorconvert_union_type()eval_forward_ref()format_type_string()get_args()get_origin()infer_type()infer_type_string()is_new_type()is_type()is_type_dict()is_typed_dict()isoftype()issubtype()normalize()optional_all()optional_any()parse_type_dict()parse_type_string()register_type()replace_class_names()strtype()to_float_or_default()to_type_dict()to_type_string()verify_required_schema()
- unitxt.types module
- unitxt.utils module
DistStubDistributionNotFoundLRUCacheLongStringSingletonVersionConflictartifacts_json_cache()deep_copy()flatten_dict()import_module_from_file()is_module_available()is_package_installed()json_dump()load_json()lru_cache_decorator()recursive_copy()recursive_deep_copy()recursive_shallow_copy()remove_numerics_and_quoted_texts()require()retry_connection_with_exponential_backoff()safe_eval()save_to_file()shallow_copy()
- unitxt.validate module
- unitxt.version module
- Module contents