Examples ✨¶
Here you will find complete coding samples showing how to perform different tasks using Unitxt. Each example comes with a self contained python file that you can run and later modify.
Basic Usage¶
Evaluate an existing dataset from the Unitxt catalog (No installation)¶
This example demonstrates how to evaluate an existing entailment dataset (wnli) using HuggingFace Datasets and Evaluate APIs, with no installation required.
Related documentation: Evaluating datasets, WNLI dataset card in catalog, Relation template in catalog.
Evaluate an existing dataset from the Unitxt catalog (with Unitxt installation)¶
Demonstrates how to evaluate an existing entailment dataset (wnli) using Unitxt native APIs. This approach is faster than using Huggingface APIs.
Related documentation: Installation , WNLI dataset card in catalog, Relation template in catalog.
Evaluate a custom dataset¶
This example demonstrates how to evaluate a user QA answering dataset in a standalone file using a user-defined task and template.
Related documentation: Add new dataset tutorial.
Evaluate a custom dataset - reusing existing catalog assets¶
This example demonstrates how to evaluate a user QA dataset using the predefined open qa task and templates. It also shows how to use preprocessing steps to align the raw input of the dataset with the predefined task fields.
Related documentation: Add new dataset tutorial, Open QA task in catalog, Open QA template in catalog.
Evaluate the impact of different templates and in-context learning demonstrations¶
This example demonstrates how different templates and the number of in-context learning examples impacts the performance of a model on an entailment task. It also shows how to register assets into a local catalog and reuse them.
Related documentation: Templates tutorial, Formatting tutorial, Using the Catalog.
Evaluate the impact of different formats and system prompts¶
This example demonstrates how different formats and system prompts affect the input provided to a llama3 chat model and evaluate their impact on the obtained scores.
Related documentation: Formatting tutorial.
Evaluate the impact of different demonstration example selections¶
This example demonstrates how different methods of selecting the demonstrations in in-context learning affect the results. Three methods are considered: fixed selection of example demonstrations for all test instances, random selection of example demonstrations for each test instance, and choosing the demonstration examples most (lexically) similar to each test instance.
Related documentation: Formatting tutorial.
Evaluate dataset with a pool of templates and some number of demonstrations¶
This example demonstrates how to evaluate a dataset using a pool of templates and a varying number of in-context learning demonstrations. It shows how to sample a template and specify the number of demonstrations for each instance from predefined lists.
Related documentation: Templates tutorial, Formatting tutorial, Using the Catalog.
Construct a benchmark of multiple datasets and obtain the final score¶
This example shows how to construct a benchmark that includes multiple datasets, each with a specific template. It demonstrates how to use these templates to evaluate the datasets and aggregate the results to obtain a final score. This approach provides a comprehensive evaluation across different tasks and datasets.
Related documentation: Benchmarks tutorial, Formatting tutorial, Using the Catalog.
LLM as Judges¶
Evaluate an existing dataset using a predefined LLM as judge¶
This example demonstrates how to evaluate an existing QA dataset (squad) using the HuggingFace Datasets and Evaluate APIs and leveraging a predefine LLM as a judge metric.
Related documentation: Evaluating datasets, LLM as a Judge Metrics Guide.
Evaluate a custom dataset using a custom LLM as Judge¶
This example demonstrates how to evaluate a user QA answering dataset in a standalone file using a user-defined task and template. In addition, it shows how to define an LLM as a judge metric, specify the template it uses to produce the input to the judge, and select the judge model and platform.
Related documentation: LLM as a Judge Metrics Guide.
Evaluate an existing dataset from the catalog comparing two custom LLM as judges¶
This example demonstrates how to evaluate a document summarization dataset by defining an LLM as a judge metric, specifying the template it uses to produce the input to the judge, and selecting the judge model and platform. The example adds two LLM judges, one that uses the ground truth (references) from the dataset and one that does not.
Related documentation: LLM as a Judge Metrics Guide.
Evaluate the quality of an LLM as judge¶
This example demonstrates how to evaluate an LLM as judge by checking its scores using the gold references of a dataset. It checks if the judge consistently prefers correct outputs over clearly wrong ones. Note that to check the the ability of the LLM as judge to discern suitable differences between partially correct answers requires more refined tests and corresponding labeled data. The example shows an 8b llama based judge is not a good judge for a summarization task, while the 70b model performs much better.
Related documentation: LLM as a Judge Metrics Guide.
Evaluate your model on the Arena Hard benchmark using a custom LLMaJ¶
This example demonstrates how to evaluate a user model on the Arena Hard benchmark, using an LLMaJ other than the GPT4.
Related documentation: Evaluate a Model on Arena Hard Benchmark.
Evaluate a judge model performance judging the Arena Hard Benchmark¶
This example demonstrates how to evaluate the capabilities of a user model, to act as a judge on the Arena Hard benchmark. The model is evaluated on its capability to give a judgment that is in correlation with GPT4 judgment on the benchmark.
Related documentation: Evaluate a Model on Arena Hard Benchmark.
Evaluate using ensemble of LLM as a judge metrics¶
This example demonstrates how to create a metric which is an ensemble of LLM as a judge metrics. The example shows how to ensemble two judges which uses different templates.
Related documentation: LLM as a Judge Metrics Guide.
Evaluate predictions of models using pre-trained ensemble of LLM as judges¶
This example demonstrates how to use a pre-trained ensemble model or an off-the-shelf LLM as judge to assess multi-turn conversation quality of models on a set of pre-defined metrics.
Topicality: Response of the model only contains information that is related to and helpful for the user inquiry. Example code <https://github.com/IBM/unitxt/blob/main/examples/evaluate_ensemble_judge.py>
Groundedness: Every substantial claim in the response of the model is derivable from the content of the document Example code <https://github.com/IBM/unitxt/blob/main/examples/evaluate_grounded_ensemble_judge.py>
IDK: Does the model response say I don’t know? Example code <https://github.com/IBM/unitxt/blob/main/examples/evaluate_idk_judge.py>
Related documentation: LLM as a Judge Metrics Guide.
RAG¶
Evaluate RAG response generation¶
This example demonstrates how to use the standard Unitxt RAG response generation task. The response generation task is the following: Given a question and one or more context(s), generate an answer that is correct and faithful to the context(s). The example shows how to map the dataset input fields to the RAG response task fields and use the existing metrics to evaluate model results.
Related documentation: RAG Guide. Response generation task.
Multi-Modality¶
Evaluate Image-Text to Text Model¶
This example demonstrates how to evaluate an image-text to text model using Unitxt. The task involves generating text responses based on both image and text inputs. This is particularly useful for tasks like visual question answering (VQA) where the model needs to understand and reason about visual content to answer questions. The example shows how to:
Load a pre-trained image-text model (LLaVA in this case)
Prepare a dataset with image-text inputs
Run inference on the model
Evaluate the model’s predictions
The code uses the document VQA dataset in English, applies a QA template with context, and formats it for the LLaVA model. It then selects a subset of the test data, generates predictions, and evaluates the results. This approach can be adapted for various image-text to text tasks, such as image captioning, visual reasoning, or multimodal dialogue systems.
Related documentation: Multi-Modality Guide.
Evaluate Image-Text to Text Model With Different Templates¶
Evaluate Image-Text to Text Models with different templates and explore the sensitivity of the model to different textual variations. Example code
Related documentation: Multi-Modality Guide.
Types and Serializers¶
Custom Types and Serializers¶
This example show how to define new data types as well as the way these data type should be handled when processed to text.
Related documentation: Types and Serializers Guide.