Unitxt is a Python library for enterprise-grade evaluation of AI performance, offering the world's largest catalog of tools and data for end-to-end AI benchmarking.

How would you like to start?

Evaluate models on existing tasks and data

Evaluate standard tasks with my data

Create a Benchmark from Existing Datasets

Evaluate with special processing or metrics

Craft and use LLMs as a Judges

Evaluate different modalities and data types

Why Unitxt?

Unitxt was built by IBM Research to host a maintainable large collection of evaluation assets. If you care about robust evaluation that last, then Unitxt is for you.

IBM

Research

Unitxt catalog is a one stop shop containing well documented assets constructing robust evaluation pipelines such as: Task Instructions, Data Loaders, Data Types Serializers and Inference Engines

Evaluation Tasks

64

LLM Ready
Datasets

3,174

Prompts

342

Metrics

462

Custom
Benchmarks

6

End to End evaluation made simple

unitxt_example.py

 from unitxt import evaluate, create_dataset
 from unitxt.blocks import Task, InputOutputTemplate
 from unitxt.inference import HFAutoModelInferenceEngine

 # Question-answer dataset
 data = [
     {"question": "What is the capital of Texas?", "answer": "Austin"},
     {"question": "What is the color of the sky?", "answer": "Blue"},
 ]

 # Define the task and evaluation metric
 task = Task(
     input_fields={"question": str},
     reference_fields={"answer": str},
     prediction_type=str,
     metrics=["metrics.accuracy"],
 )

 # Create a template to format inputs and outputs
 template = InputOutputTemplate(
     instruction="Answer the following question.",
     input_format="{question}",
     output_format="{answer}",
     postprocessors=["processors.lower_case"],
 )

 # Prepare the dataset
 dataset = create_dataset(
     task=task,
     template=template,
     format="formats.chat_api",
     test_set=data,
     split="test",
 )

 # Set up the model (supports Hugging Face, WatsonX, OpenAI, etc.)
 model = HFAutoModelInferenceEngine(
     model_name="Qwen/Qwen1.5-0.5B-Chat", max_new_tokens=32
 )

 # Generate predictions and evaluate
 predictions = model(dataset)
 results = evaluate(predictions=predictions, data=dataset)

 # Print results
 print("Global Results:\n", results.global_scores.summary)
 print("Instance Results:\n", results.instance_scores.summary)
      

End to End evaluation made simple

Welcome!¶