Code Documentation¶
unitxt package¶
Subpackages¶
- unitxt.assistant package
- unitxt.service package
- unitxt.test_utils package
- unitxt.ui package
Submodules¶
- unitxt.api module
- unitxt.artifact module
AbstractCatalogArtifactArtifact.from_dict()Artifact.get_artifact_type()Artifact.get_pretty_print_name()Artifact.is_artifact_dict()Artifact.is_artifact_file()Artifact.is_possible_identifier()Artifact.is_registered_class()Artifact.is_registered_class_name()Artifact.is_registered_type()Artifact.load()Artifact.prepare()Artifact.prepare_args()Artifact.process_data_after_load()Artifact.process_data_before_dump()Artifact.register_class()Artifact.save()Artifact.serialize()Artifact.to_json()Artifact.verify()Artifact.verify_artifact_dict()Artifact.verify_data_classification_policy()Artifact.verify_instance()
ArtifactLinkArtifactListCatalogsMissingArtifactTypeErrorUnitxtArtifactNotFoundErrorUnrecognizedArtifactTypeErrordict_diff_string()fetch_artifact()get_artifacts_data_classification()get_catalog_name_and_args()get_closest_artifact_type()get_raw()is_name_legal_for_catalog()maybe_recover_artifact()maybe_recover_artifacts_structure()register_all_artifacts()reset_artifacts_json_cache()verbosed_fetch_artifact()verify_legal_catalog_name()
- unitxt.augmentors module
- unitxt.benchmark module
- unitxt.blocks module
- unitxt.card module
- unitxt.catalog module
- unitxt.collections module
- unitxt.collections_operators module
- unitxt.dataclass module
AbstractFieldAbstractFieldErrorDataclassDataclassMetaFieldFinalFieldFinalFieldErrorInternalFieldMissingDefaultErrorNonPositionalFieldOptionalFieldRequiredFieldRequiredFieldErrorTypeMismatchErrorUndefinedUnexpectedArgumentErrorabstract_fields()asdict()class_fields()external_fields_names()fields()fields_names()final_fields()get_field_default()get_fields()is_abstract_field()is_class_method()is_dataclass()is_final_field()is_possible_field()required_fields()
- unitxt.dataset module
- unitxt.dataset_utils module
- unitxt.deprecation_utils module
- unitxt.dialog_operators module
- unitxt.dict_utils module
- unitxt.error_utils module
- unitxt.eval_utils module
- unitxt.file_utils module
- unitxt.formats module
- unitxt.fusion module
- unitxt.generator_utils module
- unitxt.hf_utils module
- unitxt.image_operators module
- unitxt.inference module
AsyncTokenBucketAzureOpenAIInferenceEngineCredentialsOpenAiCrossProviderInferenceEngineGenericInferenceEngineHFAutoModelInferenceEngineHFGenerationParamsMixinHFInferenceEngineBaseHFLlavaInferenceEngineHFOptionSelectingInferenceEngineHFPeftInferenceEngineHFPipelineBasedInferenceEngineIbmGenAiInferenceEngineIbmGenAiInferenceEngineParamsMixinInferenceEngineLMMSEvalBaseInferenceEngineLMMSEvalInferenceEngineLMMSEvalLoglikelihoodInferenceEngineLazyLoadMixinListWithMetadataLiteLLMInferenceEngineLogProbInferenceEngineMockInferenceEngineMockModeMixinOllamaInferenceEngineOpenAiInferenceEngineOpenAiInferenceEngineParamsMixinOptionSelectingByLogProbsInferenceEngineRITSInferenceEngineStandardAPIParamsMixinTextGenerationInferenceOutputTogetherAiInferenceEngineTogetherAiInferenceEngineParamsMixinTorchDeviceMixinVLLMInferenceEngineVLLMParamsMixinVLLMRemoteInferenceEngineWMLChatParamsMixinWMLGenerationParamsMixinWMLInferenceEngineBaseWMLInferenceEngineChatWMLInferenceEngineGenerationbatched()get_images_without_text()get_model_and_label_id()get_text_without_images()mock_logprobs_default_value_factory()run_with_imap()
- unitxt.instructions module
- unitxt.llm_as_judge module
- unitxt.llm_as_judge_chat_templates module
- unitxt.llm_as_judge_constants module
CriteriaCriteriaOptionCriteriaWithOptionsDirectCriteriaCatalogEnumDirectCriteriaCatalogEnum.ANSWER_RELEVANCEDirectCriteriaCatalogEnum.ANSWER_RELEVANCE_ASSISTANT_MESSAGEDirectCriteriaCatalogEnum.COHERENCEDirectCriteriaCatalogEnum.CONCISENESSDirectCriteriaCatalogEnum.CONSISTENCYDirectCriteriaCatalogEnum.CONTEXT_RELEVANCE_CONTEXTDirectCriteriaCatalogEnum.CONVERSATIONALDirectCriteriaCatalogEnum.CORRECTNESS_BASED_ON_GROUND_TRUTHDirectCriteriaCatalogEnum.EFFECTIVENESSDirectCriteriaCatalogEnum.EMPATHYDirectCriteriaCatalogEnum.ENGAGEMENTDirectCriteriaCatalogEnum.EXAMPLES_AND_DETAILSDirectCriteriaCatalogEnum.FLUENCYDirectCriteriaCatalogEnum.GENERAL_HARM_ASSISTANT_MESSAGEDirectCriteriaCatalogEnum.GENERAL_HARM_USER_MESSAGEDirectCriteriaCatalogEnum.GRAMMAR_AND_PUNCTUATIONDirectCriteriaCatalogEnum.GROUNDEDNESS_ASSISTANT_MESSAGEDirectCriteriaCatalogEnum.HARMFULNESSDirectCriteriaCatalogEnum.INFORMATION_FROM_REFERENCEDirectCriteriaCatalogEnum.INFORMATION_OUTSIDE_REFERENCEDirectCriteriaCatalogEnum.INSENSITIVITYDirectCriteriaCatalogEnum.IRRELEVANT_INFORMATIONDirectCriteriaCatalogEnum.JAILBREAK_USER_MESSAGEDirectCriteriaCatalogEnum.MANIPULATIVEDirectCriteriaCatalogEnum.NATURALNESSDirectCriteriaCatalogEnum.OBJECTIVITYDirectCriteriaCatalogEnum.PROFANITY_ASSISTANT_MESSAGEDirectCriteriaCatalogEnum.PROFANITY_USER_MESSAGEDirectCriteriaCatalogEnum.PROFESSIONAL_TONEDirectCriteriaCatalogEnum.QUALITYDirectCriteriaCatalogEnum.REFERENCE_DOCUMENT_FAITHFULNESSDirectCriteriaCatalogEnum.RELEVANCEDirectCriteriaCatalogEnum.SOCIAL_BIAS_ASSISTANT_MESSAGEDirectCriteriaCatalogEnum.SOCIAL_BIAS_USER_MESSAGEDirectCriteriaCatalogEnum.STRUCTUREDirectCriteriaCatalogEnum.SUMMARIZATION_PREFERENCEDirectCriteriaCatalogEnum.TEMPERATUREDirectCriteriaCatalogEnum.TRUTHFULNESSDirectCriteriaCatalogEnum.UNETHICAL_BEHAVIOR_ASSISTANT_MESSAGEDirectCriteriaCatalogEnum.UNETHICAL_BEHAVIOR_USER_MESSAGEDirectCriteriaCatalogEnum.VIOLENCE_ASSISTANT_MESSAGEDirectCriteriaCatalogEnum.VIOLENCE_USER_MESSAGE
EvaluatorMetadataEvaluatorNameEnumEvaluatorTypeEnumModelProviderEnumOptionSelectionStrategyEnumPairwiseCriteriaCatalogEnumPairwiseCriteriaCatalogEnum.EMAIL_INCLUSIVITYPairwiseCriteriaCatalogEnum.FACTUALLY_CONSISTENTPairwiseCriteriaCatalogEnum.FUNNY_JOKEPairwiseCriteriaCatalogEnum.INCLUSIVITYPairwiseCriteriaCatalogEnum.REFERENCE_DOCUMENT_FAITHFULNESSPairwiseCriteriaCatalogEnum.SUMMARIZATION_PREFERENCEPairwiseCriteriaCatalogEnum.TEMPERATURE
- unitxt.llm_as_judge_from_template module
LLMAsJudgeLLMAsJudgeBaseLLMAsJudgeBase.main_scoreLLMAsJudgeBase.taskLLMAsJudgeBase.templateLLMAsJudgeBase.formatLLMAsJudgeBase.system_promptLLMAsJudgeBase.inference_modelLLMAsJudgeBase.reduction_mapLLMAsJudgeBase.batch_sizeLLMAsJudgeBase.get_metric_results_from_prediction_outputs()LLMAsJudgeBase.infer_instances()LLMAsJudgeBase.prediction_typeLLMAsJudgeBase.prepare_instances()
TaskBasedLLMasJudgeget_task_data_dict()
- unitxt.llm_as_judge_operators module
- unitxt.llm_as_judge_utils module
- unitxt.loaders module
- unitxt.logging_utils module
- unitxt.metric module
- unitxt.metric_utils module
DeleteTargetPrefixEvaluationResultsFromPredictionsAndOriginalDataGlobalScoresGroupsScoresInstanceInputInstanceScoresJoinSubsetsAndGroupsMetricRecipeMetricRequestMetricResponsePostProcessRecipeSplitSubsetsAndGroupsSubsetsScoresget_remote_metrics_endpoint()get_remote_metrics_names()group_str()group_str_to_key_value()nan_mean()stream_name_to_origin_subset_group()
- unitxt.metrics module
ANLSAccuracyAccuracyFastAggregationReductionBertScoreBinaryAccuracyBinaryMaxAccuracyBinaryMaxF1BulkInstanceMetricCharEditDistanceCharEditDistanceAccuracyConfidenceIntervalMixinCustomF1CustomF1FuzzyDetectorDictReductionEvaluationInputExactMatchMMF1F1BinaryF1BinaryPosOnlyF1FastF1MacroF1MacroMultiLabelF1MicroF1MicroMultiLabelF1MultiLabelF1StringsF1WeightedFaithfulnessHHEMFinQAEvalFixedGroupAbsvalNormCohensHParaphraseAccuracyFixedGroupAbsvalNormCohensHParaphraseStringContainmentFixedGroupAbsvalNormHedgesGParaphraseAccuracyFixedGroupAbsvalNormHedgesGParaphraseStringContainmentFixedGroupMeanAccuracyFixedGroupMeanBaselineAccuracyFixedGroupMeanBaselineStringContainmentFixedGroupMeanParaphraseAccuracyFixedGroupMeanParaphraseStringContainmentFixedGroupMeanStringContainmentFixedGroupNormCohensHParaphraseAccuracyFixedGroupNormCohensHParaphraseStringContainmentFixedGroupNormHedgesGParaphraseAccuracyFixedGroupNormHedgesGParaphraseStringContainmentFixedGroupPDRParaphraseAccuracyFixedGroupPDRParaphraseStringContainmentFuzzyNerGlobalMetricGraniteGuardianAgenticRiskGraniteGuardianAssistantRiskGraniteGuardianBaseGraniteGuardianCustomRiskGraniteGuardianRagRiskGraniteGuardianUserRiskGroupMeanAccuracyGroupMeanStringContainmentGroupMeanTokenOverlapHuggingfaceBulkMetricHuggingfaceInstanceMetricHuggingfaceMetricInstanceMetricIsCodeMixedJaccardIndexKPAKendallTauMetricKeyValueExtractionLlamaIndexCorrectnessLlamaIndexFaithfulnessLlamaIndexLLMMetricMAPMRRMapReduceMetricMatthewsCorrelationMaxAccuracyMaxReductionMeanReductionMeteorMeteorFastMetricMetricPipelineMetricWithConfidenceIntervalMetricsEnsembleMetricsListNDCGNERNLTKMixinNormalizedSacrebleuPerplexityPrecisionBinaryPrecisionMacroMultiLabelPrecisionMicroMultiLabelPredictionLengthRandomForestMetricsEnsembleRecallBinaryRecallMacroMultiLabelRecallMicroMultiLabelReductionInstanceMetricRegardMetricRelaxedCorrectnessRemoteMetricRerankRecallRetrievalAtKRetrievalMetricRewardRiskTypeRocAucRougeRougeHFSQLExecutionAccuracySQLNonExecutionAccuracySafetyMetricSentenceBertSpearmanrSquadStatisticStringContainmentStringContainmentRatioTokenOverlapUnsortedListExactMatchUpdateStreamWebsrcSquadF1WeightedWinRateCorrelationWerabstract_factory()abstract_field()get_index_or_default()hf_evaluate_load()interpret_effect_size()is_original_key()mean_subgroup_score()nan_max()nan_mean()nan_std()new_random_generator()normalize_answer()normalized_cohens_h()normalized_hedges_g()parse_string_types_instead_of_actual_objects()performance_drop_rate()pytrec_eval_at_k()validate_subgroup_types()
- unitxt.normalizers module
- unitxt.operator module
InstanceOperatorInstanceOperatorValidatorInstanceOperatorWithMultiStreamAccessMissingRequirementsErrorMultiStreamOperatorOperatorOperatorErrorPackageRequirementsMixinPagedStreamOperatorSequentialMixinSequentialOperatorSequentialOperatorInitializerSideEffectOperatorSingleStreamOperatorSingleStreamReducerSourceOperatorSourceSequentialOperatorStreamInitializerOperatorStreamOperatorStreamingOperatorinstance_generator()instance_result()stream_single()
- unitxt.operators module
- Operators: Building Blocks of Unitxt Processing Pipelines
AddConstantAddIDApplyApplyMetricApplyOperatorsFieldApplyStreamOperatorsFieldArtifactFetcherMixinBalanceCastCastFieldsCollateInstancesCollateInstancesByFieldComputeExpressionMixinCopyDeduplicateDefaultPlaceHolderDeterministicBalancerDivideAllFieldsByDownloadErrorDownloadOperatorDuplicateInstancesEncodeLabelsExecuteExpressionExtractFieldValuesExtractMostCommonFieldValuesExtractZipFileFeatureGroupedShuffleFieldOperatorFillnaFilterByConditionFilterByConditionBasedOnFieldsFilterByExpressionFlattenInstancesFromIterablesGetItemByIndexIndexOfInstanceFieldOperatorInterleaveListsToDialogOperatorIntersectIntersectCorrespondingFieldsIterableSourceJoinStrLengthBalancerListFieldValuesMapInstanceValuesMapValuesMergeStreamsMinimumOneExamplePerLabelRefinerPerturbRecursiveCopyRemoveFieldsRemoveValuesRenameSelectFieldsSetShuffleShuffleFieldValuesSplitByNestedGroupSplitByValueStreamRefinerTakeByFieldUnexpectedHttpCodeErrorUniqueWikipediaFetcherZipFieldValuesupdate_scores_of_stream_instances()
- unitxt.parsing_utils module
- unitxt.processors module
AddPrefixCapitalizeConvertToBooleanDictOfListsToPairsExtractArenaHardNumericalJudgmentExtractMtBenchLabelJudgmentExtractMtBenchRatingJudgmentExtractSafeUnsafeJudgmentExtractVerbalJudgementBadGoodExtractVerbalJudgmentExtractWithRegexFirstCharacterFixWhiteSpaceGetSQLGetStringAfterInferDictsToBinaryLogprobsListToEmptyEntitiesTuplesLiteralEvalLowerLowerCaseTillPuncMatchClosestOptionPostProcessRegexParserRemoveArticlesRemovePunctuationsScaleNumberToZeroOneReturnZeroIfFailsSplitStripStanceToProConStrToFloatFormatStringEqualsSubstringTakeFirstNonEmptyLineTakeFirstWordTakeLastNonEmptyLineToListByCommaToListByCommaSpaceToStringToStringStrippedToYesOrNoneUpperYesNoToIntYesToOneElseZeroprocess_instance_value()
- unitxt.random_utils module
- unitxt.recipe module
- unitxt.register module
- unitxt.schema module
- unitxt.serializers module
DefaultListSerializerDefaultSerializerDialogSerializerDictAsJsonSerializerDocumentSerializerImageSerializerListSerializerMultiDocumentSerializerMultiTypeSerializerNumberQuantizingSerializerNumberSerializerSQLDatabaseAsSchemaSerializerSerializerSingleTypeSerializerTableSerializerVideoSerializer
- unitxt.settings_utils module
- unitxt.span_lableing_operators module
- unitxt.split_utils module
- unitxt.splitters module
- unitxt.sql_utils module
CacheDatabaseConnectorInMemoryDatabaseConnectorLocalSQLiteConnectorRemoteDatabaseConnectorcollect_clause()execute_query_local()execute_query_remote()extract_select_columns()extract_select_info()generate_cache_key()get_cache()get_db_connector()is_sqlglot_parsable()is_sqlparse_parsable()sql_exact_match()sqlglot_optimized_equivalence()sqlglot_parsed_queries_equivalent()sqlparse_queries_equivalent()strip_alias()
- unitxt.standard module
- unitxt.stream module
- unitxt.stream_operators module
- unitxt.string_operators module
- unitxt.struct_data_operators module
ConstructTableFromRowsColsConvertTableColNamesToSequentialDumpJsonDuplicateTableColumnsDuplicateTableRowsGetNumOfTableCellsInsertEmptyTableRowsJsonStrToDictListToKeyValPairsLoadJsonMapHTMLTableToJSONMapTableListsToStdTableJSONMaskColumnsNamesSerializeKeyValPairsSerializeTableSerializeTableAsConcatenationSerializeTableAsDFLoaderSerializeTableAsHTMLSerializeTableAsImageSerializeTableAsIndexedRowMajorSerializeTableAsJsonSerializeTableAsMarkdownSerializeTableRowAsListSerializeTableRowAsTextSerializeTriplesShuffleColumnsNamesShuffleTableColumnsShuffleTableRowsTransposeTableTruncateTableCellsTruncateTableRowsshuffle_columns()shuffle_rows()truncate_cell()
- unitxt.system_prompts module
- unitxt.task module
- unitxt.templates module
ApplyRandomTemplateApplySingleTemplateApplyTemplateDialogFieldsDataDialogPairwiseChoiceTemplateDialogTemplateInputFormatTemplateInputOutputTemplateInputOutputTemplateWithCustomTargetJsonOutputFormatTemplateJsonOutputTemplateKeyValTemplateMultiLabelTemplateMultiReferenceTemplateMultipleChoiceTemplateNullTemplateOutputFormatTemplateOutputQuantizingTemplatePairwiseChoiceTemplatePairwiseComparativeRatingTemplateSpanLabelingBaseTemplateSpanLabelingJsonTemplateSpanLabelingTemplateTemplateTemplateFormatKeyErrorTemplatesDictTemplatesListYesNoTemplateescape_chars()random()
- unitxt.text_utils module
- unitxt.type_utils module
GenericTypedDictNormalizedTypeUnsupportedTypeErrorconvert_union_type()eval_forward_ref()format_type_string()get_args()get_origin()infer_type()infer_type_string()is_new_type()is_type()is_type_dict()is_typed_dict()isoftype()issubtype()normalize()optional_all()optional_any()parse_type_dict()parse_type_string()register_type()replace_class_names()strtype()to_float_or_default()to_type_dict()to_type_string()verify_required_schema()
- unitxt.types module
- unitxt.utils module
LRUCacheLongStringSingletonartifacts_json_cache()deep_copy()flatten_dict()import_module_from_file()is_module_available()is_package_installed()json_dump()load_json()recursive_copy()recursive_deep_copy()recursive_shallow_copy()remove_numerics_and_quoted_texts()safe_eval()save_to_file()shallow_copy()
- unitxt.validate module
- unitxt.version module