Code Documentation¶
unitxt package¶
Subpackages¶
- unitxt.assistant package
- unitxt.service package
- unitxt.test_utils package
- unitxt.ui package
- Submodules
- unitxt.ui.gradio_utils module
- unitxt.ui.load_catalog_data module
check_augmentable()create_temporary_dict()get_catalog_dirs()get_catalog_items()get_catalog_items_from_dir()get_file_from_item_name()get_templates()get_templates_from_dict()get_templates_from_list()get_templates_from_str()load_cards_data()register_existing_template()register_inline_template()safe_load_json()save_temporary_template()
- unitxt.ui.run module
- unitxt.ui.settings module
- unitxt.ui.ui_tester module
- unitxt.ui.ui_utils module
- Module contents
- Submodules
Submodules¶
- unitxt.api module
- unitxt.artifact module
AbstractCatalogArtifactArtifact.from_dict()Artifact.get_artifact_type()Artifact.get_pretty_print_name()Artifact.is_artifact_dict()Artifact.is_artifact_file()Artifact.is_possible_identifier()Artifact.is_registered_class()Artifact.is_registered_class_name()Artifact.is_registered_type()Artifact.load()Artifact.prepare()Artifact.prepare_args()Artifact.process_data_after_load()Artifact.process_data_before_dump()Artifact.register_class()Artifact.save()Artifact.serialize()Artifact.to_json()Artifact.to_yaml()Artifact.verify()Artifact.verify_artifact_dict()Artifact.verify_data_classification_policy()Artifact.verify_instance()
ArtifactLinkArtifactListCatalogsMissingArtifactTypeErrorUnitxtArtifactNotFoundErrorUnrecognizedArtifactTypeErrordict_diff_string()fetch_artifact()get_artifacts_data_classification()get_catalog_name_and_args()get_closest_artifact_type()get_raw()is_name_legal_for_catalog()maybe_recover_artifact()maybe_recover_artifacts_structure()register_all_artifacts()reset_artifacts_json_cache()verbosed_fetch_artifact()verify_legal_catalog_name()
- unitxt.augmentors module
- unitxt.base_metric module
- unitxt.benchmark module
- unitxt.blocks module
- unitxt.card module
- unitxt.catalog module
- unitxt.collections module
- unitxt.collections_operators module
- unitxt.dataclass module
AbstractFieldAbstractFieldErrorDataclassDataclassMetaFieldFinalFieldFinalFieldErrorInternalFieldMissingDefaultErrorNonPositionalFieldOptionalFieldRequiredFieldRequiredFieldErrorTypeMismatchErrorUndefinedUnexpectedArgumentErrorabstract_fields()asdict()class_fields()external_fields_names()fields()fields_names()final_fields()get_field_default()get_fields()is_abstract_field()is_class_method()is_dataclass()is_final_field()is_possible_field()required_fields()to_dict()
- unitxt.dataset module
- unitxt.dataset_utils module
- unitxt.deprecation_utils module
- unitxt.dialog_operators module
- unitxt.dict_utils module
- unitxt.error_utils module
- unitxt.eval_utils module
- unitxt.evaluate_cli module
cli_load_dataset()configure_unitxt_settings()extract_overwrite_args()extract_scores()initialize_inference_engine()main()prepare_kwargs()prepare_output_paths()prepend_timestamp_to_path()process_and_save_results()run_evaluation()run_inference()setup_logging()setup_parser()summarize_cli()try_parse_json()
- unitxt.file_utils module
- unitxt.formats module
- unitxt.fusion module
- unitxt.generator_utils module
- unitxt.hf_utils module
- unitxt.image_operators module
- unitxt.inference module
AsyncTokenBucketAzureOpenAIInferenceEngineCredentialsOpenAiCrossProviderInferenceEngineGenericInferenceEngineHFAutoModelInferenceEngineHFGenerationParamsMixinHFInferenceEngineBaseHFLlavaInferenceEngineHFOptionSelectingInferenceEngineHFPeftInferenceEngineHFPipelineBasedInferenceEngineInferenceEngineLMMSEvalBaseInferenceEngineLMMSEvalInferenceEngineLMMSEvalLoglikelihoodInferenceEngineLazyLoadMixinListWithMetadataLiteLLMInferenceEngineLogProbInferenceEngineMetricInferenceEngineMockInferenceEngineMockModeMixinOllamaInferenceEngineOpenAiInferenceEngineOpenAiInferenceEngineParamsMixinOptionSelectingByLogProbsInferenceEngineRITSInferenceEngineStandardAPIParamsMixinTextGenerationInferenceOutputTogetherAiInferenceEngineTogetherAiInferenceEngineParamsMixinTorchDeviceMixinVLLMInferenceEngineVLLMParamsMixinVLLMRemoteInferenceEngineWMLChatParamsMixinWMLGenerationParamsMixinWMLInferenceEngineBaseWMLInferenceEngineChatWMLInferenceEngineGenerationbatched()get_images_without_text()get_model_and_label_id()get_text_without_images()mock_logprobs_default_value_factory()run_with_imap()
- unitxt.instructions module
- unitxt.llm_as_judge module
- unitxt.llm_as_judge_chat_templates module
- unitxt.llm_as_judge_constants module
CriteriaCriteriaOptionCriteriaWithOptionsDirectCriteriaCatalogEnumDirectCriteriaCatalogEnum.ANSWER_RELEVANCEDirectCriteriaCatalogEnum.ANSWER_RELEVANCE_ASSISTANT_MESSAGEDirectCriteriaCatalogEnum.COHERENCEDirectCriteriaCatalogEnum.COMPLIANCE_ASSISTANT_MESSAGEDirectCriteriaCatalogEnum.CONCISENESSDirectCriteriaCatalogEnum.CONSISTENCYDirectCriteriaCatalogEnum.CONTEXT_RELEVANCE_CONTEXTDirectCriteriaCatalogEnum.CONVERSATIONALDirectCriteriaCatalogEnum.CORRECTNESS_BASED_ON_GROUND_TRUTHDirectCriteriaCatalogEnum.EFFECTIVENESSDirectCriteriaCatalogEnum.EMPATHYDirectCriteriaCatalogEnum.ENGAGEMENTDirectCriteriaCatalogEnum.EXAMPLES_AND_DETAILSDirectCriteriaCatalogEnum.FLUENCYDirectCriteriaCatalogEnum.GENERAL_HARM_ASSISTANT_MESSAGEDirectCriteriaCatalogEnum.GENERAL_HARM_USER_MESSAGEDirectCriteriaCatalogEnum.GRAMMAR_AND_PUNCTUATIONDirectCriteriaCatalogEnum.GROUNDEDNESS_ASSISTANT_MESSAGEDirectCriteriaCatalogEnum.HARMFULNESSDirectCriteriaCatalogEnum.INFORMATION_FROM_REFERENCEDirectCriteriaCatalogEnum.INFORMATION_OUTSIDE_REFERENCEDirectCriteriaCatalogEnum.INSENSITIVITYDirectCriteriaCatalogEnum.IRRELEVANT_INFORMATIONDirectCriteriaCatalogEnum.JAILBREAK_USER_MESSAGEDirectCriteriaCatalogEnum.LOGICAL_VALIDITY_OF_REASONINGDirectCriteriaCatalogEnum.MANIPULATIVEDirectCriteriaCatalogEnum.NATURALNESSDirectCriteriaCatalogEnum.OBJECTIVITYDirectCriteriaCatalogEnum.PROFANITY_ASSISTANT_MESSAGEDirectCriteriaCatalogEnum.PROFANITY_USER_MESSAGEDirectCriteriaCatalogEnum.PROFESSIONAL_TONEDirectCriteriaCatalogEnum.QUALITYDirectCriteriaCatalogEnum.REFERENCE_DOCUMENT_FAITHFULNESSDirectCriteriaCatalogEnum.RELEVANCEDirectCriteriaCatalogEnum.SOCIAL_BIAS_ASSISTANT_MESSAGEDirectCriteriaCatalogEnum.SOCIAL_BIAS_USER_MESSAGEDirectCriteriaCatalogEnum.STEP_BY_STEP_BAD_FINAL_ANSWERDirectCriteriaCatalogEnum.STEP_BY_STEP_NON_COHERENTDirectCriteriaCatalogEnum.STEP_BY_STEP_REASONING_ARITHMETICDirectCriteriaCatalogEnum.STEP_BY_STEP_REASONING_BAD_GRAMMARDirectCriteriaCatalogEnum.STEP_BY_STEP_REASONING_COHERENCYDirectCriteriaCatalogEnum.STEP_BY_STEP_REASONING_COMMONSENSEDirectCriteriaCatalogEnum.STEP_BY_STEP_REASONING_CONTRADICTIONDirectCriteriaCatalogEnum.STEP_BY_STEP_REASONING_HALLUCINATIONDirectCriteriaCatalogEnum.STEP_BY_STEP_REASONING_MISSING_STEPSDirectCriteriaCatalogEnum.STEP_BY_STEP_REASONING_NON_FACTUALDirectCriteriaCatalogEnum.STEP_BY_STEP_REASONING_OVERALL_QUALITYDirectCriteriaCatalogEnum.STEP_BY_STEP_REASONING_REDUNDANCYDirectCriteriaCatalogEnum.STEP_BY_STEP_REASONING_REPETITIONDirectCriteriaCatalogEnum.STRUCTUREDirectCriteriaCatalogEnum.SUMMARIZATION_COHERENCEDirectCriteriaCatalogEnum.SUMMARIZATION_FLUENCYDirectCriteriaCatalogEnum.SUMMARIZATION_INFORMATIVENESSDirectCriteriaCatalogEnum.SUMMARIZATION_PREFERENCEDirectCriteriaCatalogEnum.SUMMARIZATION_RELEVANCEDirectCriteriaCatalogEnum.TEMPERATUREDirectCriteriaCatalogEnum.TOXICITYDirectCriteriaCatalogEnum.TRANSLATION_QUALITYDirectCriteriaCatalogEnum.TRUTHFULNESSDirectCriteriaCatalogEnum.UNETHICAL_BEHAVIOR_ASSISTANT_MESSAGEDirectCriteriaCatalogEnum.UNETHICAL_BEHAVIOR_USER_MESSAGEDirectCriteriaCatalogEnum.VIOLENCE_ASSISTANT_MESSAGEDirectCriteriaCatalogEnum.VIOLENCE_USER_MESSAGE
EvaluatorMetadataEvaluatorNameEnumEvaluatorTypeEnumModelProviderEnumOptionSelectionStrategyEnumPairwiseCriteriaCatalogEnumPairwiseCriteriaCatalogEnum.EMAIL_INCLUSIVITYPairwiseCriteriaCatalogEnum.FACTUALLY_CONSISTENTPairwiseCriteriaCatalogEnum.FUNNY_JOKEPairwiseCriteriaCatalogEnum.INCLUSIVITYPairwiseCriteriaCatalogEnum.REFERENCE_DOCUMENT_FAITHFULNESSPairwiseCriteriaCatalogEnum.SUMMARIZATION_PREFERENCEPairwiseCriteriaCatalogEnum.TEMPERATURE
get_likert_scale_criteria()get_yes_no_criteria()
- unitxt.llm_as_judge_from_template module
LLMAsJudgeLLMAsJudgeBaseLLMAsJudgeBase.main_scoreLLMAsJudgeBase.taskLLMAsJudgeBase.templateLLMAsJudgeBase.formatLLMAsJudgeBase.system_promptLLMAsJudgeBase.inference_modelLLMAsJudgeBase.reduction_mapLLMAsJudgeBase.batch_sizeLLMAsJudgeBase.get_metric_results_from_prediction_outputs()LLMAsJudgeBase.infer_instances()LLMAsJudgeBase.prediction_typeLLMAsJudgeBase.prepare_instances()
TaskBasedLLMasJudgeget_task_data_dict()
- unitxt.llm_as_judge_operators module
- unitxt.llm_as_judge_utils module
- unitxt.loaders module
- Loaders: Generators of Unitxt Multistreams from existing date sources
LazyLoaderLoadCSVLoadFromAPILoadFromDictionaryLoadFromHFSpaceLoadFromIBMCloudLoadFromKaggleLoadFromSklearnLoadHFLoadIOBLoadJsonFileLoadWithPandasLoaderMissingKaggleCredentialsErrorMultipleSourceLoaderTURLColumnTypeAnnotationLoaderUnitxtUnverifiedCodeErrorhf_get_dataset_splits()hf_load_dataset()read_file()
- unitxt.logging_utils module
- unitxt.metric module
- unitxt.metric_utils module
DeleteTargetPrefixEmptyPredictionEvaluationResultsFromPredictionsAndOriginalDataGlobalScoresGroupsScoresInstanceInputInstanceScoresJoinSubsetsAndGroupsMetricRecipeMetricRequestMetricResponsePostProcessRecipeSplitSubsetsAndGroupsSubsetsScoresempty_predictions_generator()get_remote_metrics_endpoint()get_remote_metrics_names()group_str()group_str_to_key_value()nan_mean()stream_name_to_origin_subset_group()
- unitxt.metrics module
ANLSAccuracyAccuracyFastAggregationReductionBertScoreBinaryAccuracyBinaryMaxAccuracyBinaryMaxF1BulkInstanceMetricCharEditDistanceCharEditDistanceAccuracyConfidenceIntervalMixinCorrelationMetricCustomF1CustomF1FuzzyDetectorDictReductionEvaluationInputExactMatchMMF1F1BinaryF1BinaryPosOnlyF1FastF1MacroF1MacroMultiLabelF1MicroF1MicroMultiLabelF1MultiLabelF1StringsF1WeightedFaithfulnessHHEMFinQAEvalFixedGroupAbsvalNormCohensHParaphraseAccuracyFixedGroupAbsvalNormCohensHParaphraseStringContainmentFixedGroupAbsvalNormHedgesGParaphraseAccuracyFixedGroupAbsvalNormHedgesGParaphraseStringContainmentFixedGroupMeanAccuracyFixedGroupMeanBaselineAccuracyFixedGroupMeanBaselineStringContainmentFixedGroupMeanParaphraseAccuracyFixedGroupMeanParaphraseStringContainmentFixedGroupMeanStringContainmentFixedGroupNormCohensHParaphraseAccuracyFixedGroupNormCohensHParaphraseStringContainmentFixedGroupNormHedgesGParaphraseAccuracyFixedGroupNormHedgesGParaphraseStringContainmentFixedGroupPDRParaphraseAccuracyFixedGroupPDRParaphraseStringContainmentFuzzyNerGlobalMetricGraniteGuardianAgenticRiskGraniteGuardianAssistantRiskGraniteGuardianBaseGraniteGuardianCustomRiskGraniteGuardianRagRiskGraniteGuardianUserRiskGroupMeanGroupMeanAccuracyGroupMeanStringContainmentGroupMeanTokenOverlapGroupMetricGroupReductionHuggingfaceBulkMetricHuggingfaceInstanceMetricHuggingfaceMetricInstanceMetricIsCodeMixedJaccardIndexJaccardIndexStringKPAKendallTauMetricKeyValueExtractionLlamaIndexCorrectnessLlamaIndexFaithfulnessLlamaIndexLLMMetricMAPMRRMapReduceMetricMatthewsCorrelationMaxAccuracyMaxReductionMeanReductionMeanSquaredErrorMeteorMeteorFastMetricBasedNerMetricPipelineMetricWithConfidenceIntervalMetricsEnsembleMetricsListMultiTurnMetricMultiTurnToolCallingMetricNDCGNERNLTKMixinNormalizedSacrebleuPearsonrPerplexityPrecisionBinaryPrecisionMacroMultiLabelPrecisionMicroMultiLabelPredictionLengthRandomForestMetricsEnsembleRecallBinaryRecallMacroMultiLabelRecallMicroMultiLabelReductionInstanceMetricReflectionToolCallingMetricReflectionToolCallingMetricSyntacticReflectionToolCallingMixinRegardMetricRelaxedCorrectnessRemoteMetricRerankRecallRetrievalAtKRetrievalMetricRewardRiskTypeRocAucRootMeanReductionRootMeanSquaredErrorRougeRougeHFSQLExecutionAccuracySQLExecutionLogicAccuracySQLNonExecutionAccuracySacreBleuStatsSafetyMetricSentenceBertSequentialSuccessSpearmanrSquadStatisticStringContainmentStringContainmentOldStringContainmentRatioTokenOverlapToolCallKeyValueExtractionToolCallingMetricUnsortedListExactMatchUpdateStreamWebsrcSquadF1WeightedWinRateCorrelationWerabstract_factory()abstract_field()get_index_or_default()hf_evaluate_load()interpret_effect_size()is_original_key()mean_subgroup_score()nan_max()nan_mean()nan_std()new_random_generator()normalize_answer()normalized_cohens_h()normalized_hedges_g()parse_string_types_instead_of_actual_objects()performance_drop_rate()pytrec_eval_at_k()validate_subgroup_types()
- unitxt.normalizers module
- unitxt.operator module
InstanceOperatorInstanceOperatorValidatorInstanceOperatorWithMultiStreamAccessMissingRequirementsErrorMultiStreamOperatorOperatorOperatorErrorPackageRequirementsMixinPagedStreamOperatorSequentialMixinSequentialOperatorSequentialOperatorInitializerSideEffectOperatorSingleStreamOperatorSingleStreamReducerSourceOperatorSourceSequentialOperatorStreamInitializerOperatorStreamOperatorStreamingOperatorinstance_generator()instance_result()stream_single()
- unitxt.operators module
- Operators: Building Blocks of Unitxt Processing Pipelines
AddConstantAddIDAddIncrementalIdApplyApplyMetricApplyOperatorsFieldApplyStreamOperatorsFieldArtifactFetcherMixinBalanceBytesToStringCastCastFieldsCollateInstancesCollateInstancesByFieldComputeExpressionMixinCopyDeduplicateDefaultPlaceHolderDeterministicBalancerDivideAllFieldsByDownloadErrorDownloadOperatorDuplicateInstancesEncodeLabelsExecuteExpressionExtractFieldValuesExtractMostCommonFieldValuesExtractZipFileFeatureGroupedShuffleFieldOperatorFillnaFilterByConditionFilterByConditionBasedOnFieldsFilterByExpressionFixJsonSchemaOfParameterTypesFlattenInstancesFromIterablesGetItemByIndexIndexOfInstanceFieldOperatorInterleaveListsToDialogOperatorIntersectIntersectCorrespondingFieldsIterableSourceJoinStrLengthBalancerListFieldValuesMapInstanceValuesMapValuesMergeStreamsMinimumOneExamplePerLabelRefinerMovePerturbReadFileRecursiveCopyRecursiveReplaceRemoveFieldsRemoveValuesRenameSelectFieldsSetShuffleShuffleFieldValuesSplitByNestedGroupStreamRefinerTakeByFieldUnexpectedHttpCodeErrorWikipediaFetcherZipFieldValuesrecursive_key_value_replace()update_scores_of_stream_instances()
- unitxt.parsing_utils module
- unitxt.processors module
AddPrefixCapitalizeConvertToBooleanDictOfListsToPairsExtractArenaHardNumericalJudgmentExtractHarmRatingJudgementExtractMtBenchLabelJudgmentExtractMtBenchRatingJudgmentExtractSafeUnsafeJudgmentExtractVerbalJudgementBadGoodExtractVerbalJudgmentExtractWithRegexFirstCharacterFixWhiteSpaceGetSQLGetStringAfterGroupDictWithRegexInferDictsToBinaryLogprobsListToEmptyEntitiesTuplesLiteralEvalLowerLowerCaseTillPuncMatchClosestOptionPostProcessRegexParserRemoveArticlesRemovePunctuationsScaleNumberToZeroOneReturnZeroIfFailsSplitStripStanceToProConStrToFloatFormatStringEqualsSubstringTakeFirstNonEmptyLineTakeFirstWordTakeLastNonEmptyLineToListByCommaToListByCommaSpaceToStringToStringStrippedToYesOrNoneUpperYesNoToIntYesToOneElseZeroprocess_instance_value()
- unitxt.random_utils module
- unitxt.recipe module
- unitxt.register module
- unitxt.schema module
- unitxt.serializers module
ConversationSerializerDefaultListSerializerDefaultSerializerDialogSerializerDictAsJsonSerializerDocumentSerializerImageSerializerListSerializerMultiDocumentSerializerMultiTypeSerializerNumberQuantizingSerializerNumberSerializerSQLDatabaseAsSchemaSerializerSerializerSingleTypeSerializerTableSerializerToolCallSerializerToolsSerializerVideoSerializer
- unitxt.settings_utils module
- unitxt.span_lableing_operators module
- unitxt.split_utils module
- unitxt.splitters module
- unitxt.standard module
- unitxt.stream module
- unitxt.stream_operators module
- unitxt.string_operators module
- unitxt.struct_data_operators module
ConstructTableFromRowsColsConvertTableColNamesToSequentialDumpJsonDuplicateTableColumnsDuplicateTableRowsGetNumOfTableCellsInsertEmptyTableRowsJsonStrToDictListToKeyValPairsLoadJsonMapHTMLTableToJSONMapTableListsToStdTableJSONMaskColumnsNamesMultipleToolCallPostProcessorParseCSVPythonCallProcessorSerializeKeyValPairsSerializeTableSerializeTableAsConcatenationSerializeTableAsDFLoaderSerializeTableAsHTMLSerializeTableAsImageSerializeTableAsIndexedRowMajorSerializeTableAsJsonSerializeTableAsMarkdownSerializeTableRowAsListSerializeTableRowAsTextSerializeTriplesShuffleColumnsNamesShuffleTableColumnsShuffleTableRowsToolCallPostProcessorTransposeTableTruncateTableCellsTruncateTableRowsextract_possible_json_str()shuffle_columns()shuffle_rows()truncate_cell()
- unitxt.system_prompts module
- unitxt.task module
- unitxt.templates module
ApplyRandomTemplateApplySingleTemplateApplyTemplateDialogFieldsDataDialogPairwiseChoiceTemplateDialogTemplateInputFormatTemplateInputOutputTemplateInputOutputTemplateWithCustomTargetJsonOutputFormatTemplateJsonOutputTemplateKeyValTemplateMultiLabelTemplateMultiReferenceTemplateMultiTurnTemplateMultipleChoiceTemplateNullTemplateOutputFormatTemplateOutputQuantizingTemplatePairwiseChoiceTemplatePairwiseComparativeRatingTemplateSpanLabelingBaseTemplateSpanLabelingJsonTemplateSpanLabelingTemplateTemplateTemplateFormatKeyErrorTemplatesDictTemplatesListYesNoTemplateescape_chars()random()
- unitxt.text2sql_utils module
CacheDatabaseConnectorInMemoryDatabaseConnectorLocalSQLiteConnectorRemoteDatabaseConnectorSQLExecutionResultSQLNonExecutionMetricResultcollect_clause()compare_dfs_bird_eval_logic()compare_dfs_ignore_colnames_ordered_rows()compare_dfs_ignore_colnames_subset()compare_dfs_ignore_colnames_unordered_rows()compare_result_dfs()execute_query_local()execute_query_remote()extract_select_columns()extract_select_info()extract_sql_from_text()generate_cache_key()get_cache()get_db_connector()get_sql_execution_results()is_sqlglot_parsable()is_sqlparse_parsable()replace_select_clause()run_query()sql_exact_match()sqlglot_optimized_equivalence()sqlglot_parsed_queries_equivalent()sqlparse_queries_equivalent()strip_alias()
- unitxt.text_utils module
- unitxt.type_utils module
GenericTypedDictNormalizedTypeUnsupportedTypeErrorconvert_union_type()eval_forward_ref()format_type_string()get_args()get_origin()infer_type()infer_type_string()is_new_type()is_type()is_type_dict()is_typed_dict()isoftype()issubtype()normalize()optional_all()optional_any()parse_type_dict()parse_type_string()register_type()replace_class_names()strtype()to_float_or_default()to_type_dict()to_type_string()verify_required_schema()
- unitxt.types module
- unitxt.utils module
DistStubDistributionNotFoundLRUCacheLongStringSingletonVersionConflictartifacts_json_cache()deep_copy()flatten_dict()import_module_from_file()is_module_available()is_package_installed()json_dump()load_json()lru_cache_decorator()recursive_copy()recursive_deep_copy()recursive_shallow_copy()remove_numerics_and_quoted_texts()require()retry_connection_with_exponential_backoff()safe_eval()save_to_file()shallow_copy()
- unitxt.validate module
- unitxt.version module