Examples ✨¶
Here you will find complete coding samples showing how to perform different tasks using Unitxt. Each example comes with a self contained python file that you can run and later modify.
Basic Usage¶
Evaluate an existing dataset from the Unitxt catalog (No installation)¶
This example demonstrates how to evaluate an existing entailment dataset (wnli) using HuggingFace Datasets and Evaluate APIs, with no installation required.
Related documentation: Evaluating datasets, WNLI dataset card in catalog, Relation template in catalog.
Evaluate an existing dataset from the Unitxt catalog (with Unitxt installation)¶
Demonstrates how to evaluate an existing entailment dataset (wnli) using Unitxt native APIs. This approach is faster than using Huggingface APIs.
Related documentation: Installation , WNLI dataset card in catalog, Relation template in catalog.
Evaluate a custom dataset¶
This example demonstrates how to evaluate a user QA answering dataset in a standalone file using a user-defined task and template.
Related documentation: Add new dataset tutorial.
Evaluate a custom dataset - reusing existing catalog assets¶
This example demonstrates how to evaluate a user QA dataset using the predefined open qa task and templates. It also shows how to use preprocessing steps to align the raw input of the dataset with the predefined task fields.
Related documentation: Add new dataset tutorial, Open QA task in catalog, Open QA template in catalog.
Evaluate the impact of different templates and in-context learning demonstrations¶
This example demonstrates how different templates and the number of in-context learning examples impacts the performance of a model on an entailment task. It also shows how to register assets into a local catalog and reuse them.
Related documentation: Templates tutorial, Formatting tutorial, Using the Catalog.
Evaluate the impact of different formats and system prompts¶
This example demonstrates how different formats and system prompts affect the input provided to a llama3 chat model and evaluate their impact on the obtained scores.
Related documentation: Formatting tutorial.
Evaluate the impact of different demonstration example selections¶
This example demonstrates how different methods of selecting the demonstrations in in-context learning affect the results. Three methods are considered: fixed selection of example demonstrations for all test instances, random selection of example demonstrations for each test instance, and choosing the demonstration examples most (lexically) similar to each test instance.
Related documentation: Formatting tutorial.
Evaluate dataset with a pool of templates and some number of demonstrations¶
This example demonstrates how to evaluate a dataset using a pool of templates and a varying number of in-context learning demonstrations. It shows how to sample a template and specify the number of demonstrations for each instance from predefined lists.
Related documentation: Templates tutorial, Formatting tutorial, Using the Catalog.
LLM as Judges¶
Evaluate an existing dataset using a predefined LLM as judge¶
This example demonstrates how to evaluate an existing QA dataset (squad) using the HuggingFace Datasets and Evaluate APIs and leveraging a predefine LLM as a judge metric.
Related documentation: Evaluating datasets, LLM as a Judge Metrics Guide.
Evaluate a custom dataset using a custom LLM as Judge¶
This example demonstrates how to evaluate a user QA answering dataset in a standalone file using a user-defined task and template. In addition, it shows how to define an LLM as a judge metric, specify the template it uses to produce the input to the judge, and select the judge model and platform.
Related documentation: LLM as a Judge Metrics Guide.
Evaluate an existing dataset from the catalog comparing two custom LLM as judges¶
This example demonstrates how to evaluate a document summarization dataset by defining an LLM as a judge metric, specifying the template it uses to produce the input to the judge, and selecting the judge model and platform. The example adds two LLM judges, one that uses the ground truth (references) from the dataset and one that does not.
Related documentation: LLM as a Judge Metrics Guide.
Evaluate the quality of an LLM as judge¶
This example demonstrates how to evaluate an LLM as judge by checking its scores using the gold references of a dataset. It checks if the judge consistently prefers correct outputs over clearly wrong ones. Note that to check the the ability of the LLM as judge to discern suitable differences between partially correct answers requires more refined tests and corresponding labeled data. The example shows an 8b llama based judge is not a good judge for a summarization task, while the 70b model performs much better.
Related documentation: LLM as a Judge Metrics Guide.
Evaluate your model on the Arena Hard benchmark using a custom LLMaJ¶
This example demonstrates how to evaluate a user model on the Arena Hard benchmark, using an LLMaJ other than the GPT4.
Related documentation: Evaluate a Model on Arena Hard Benchmark.
Evaluate a judge model performance judging the Arena Hard Benchmark¶
This example demonstrates how to evaluate the capabilities of a user model, to act as a judge on the Arena Hard benchmark. The model is evaluated on its capability to give a judgment that is in correlation with GPT4 judgment on the benchmark.
Related documentation: Evaluate a Model on Arena Hard Benchmark.
Evaluate using ensemble of LLM as a judge metrics¶
This example demonstrates how to create a metric which is an ensemble of LLM as a judge metrics. The example shows how to ensemble two judges which uses different templates.
Related documentation: LLM as a Judge Metrics Guide.
RAG¶
Evaluate RAG response generation¶
This example demonstrates how to use the standard Unitxt RAG response generation task. The response generation task is the following: Given a question and one or more context(s), generate an answer that is correct and faithful to the context(s). The example shows how to map the dataset input fields to the RAG response task fields and use the existing metrics to evaluate model results.
Related documentation: RAG Guide. Response generation task.