📄 Vision Full¶

benchmarks.vision_full

Benchmark(
    subsets={
        "doc_vqa_default": DatasetRecipe(
            card="cards.doc_vqa.lmms_eval",
        ),
        "info_vqa_default": DatasetRecipe(
            card="cards.info_vqa_lmms_eval",
        ),
        "chart_qa_default": DatasetRecipe(
            card="cards.chart_qa_lmms_eval",
        ),
        "ai2d_default": DatasetRecipe(
            card="cards.ai2d",
        ),
        "websrc_default": DatasetRecipe(
            card="cards.websrc",
        ),
        "doc_vqa_llama_vision_template": DatasetRecipe(
            card="cards.doc_vqa.lmms_eval",
            template="templates.qa.llama_vision.with_context.doc_vqa",
            format="formats.chat_api",
        ),
        "info_vqa_llama_vision_template": DatasetRecipe(
            card="cards.info_vqa_lmms_eval",
            template="templates.qa.llama_vision.with_context.info_vqa",
            format="formats.chat_api",
        ),
        "chart_qa_llama_vision_template": DatasetRecipe(
            card="cards.chart_qa_lmms_eval",
            template="templates.qa.llama_vision.with_context.chart_qa",
            format="formats.chat_api",
        ),
        "ai2d_llama_vision_template": DatasetRecipe(
            card="cards.ai2d",
            template="templates.qa.llama_vision.multiple_choice.with_context.ai2d",
            format="formats.chat_api",
        ),
    },
)

[source]

from unitxt.standard import DatasetRecipe

Explanation about DatasetRecipe¶

This class represents a standard recipe for data processing and preparation.

This class can be used to prepare a recipe. with all necessary steps, refiners and renderers included. It allows to set various parameters and steps in a sequential manner for preparing the recipe.

Args:

card (TaskCard):
TaskCard object associated with the recipe.

template (Template, optional):
Template object to be used for the recipe.

system_prompt (SystemPrompt, optional):
SystemPrompt object to be used for the recipe.

loader_limit (int, optional):
Specifies the maximum number of instances per stream to be returned from the loader (used to reduce loading time in large datasets)

format (SystemFormat, optional):
SystemFormat object to be used for the recipe.

metrics (List[str]):
list of catalog metrics to use with this recipe.

postprocessors (List[str]):
list of catalog processors to apply at post processing. (Not recommended to use from here)

group_by (List[Union[str, List[str]]]):
list of task_data or metadata keys to group global scores by.

train_refiner (StreamRefiner, optional):
Train refiner to be used in the recipe.

max_train_instances (int, optional):
Maximum training instances for the refiner.

validation_refiner (StreamRefiner, optional):
Validation refiner to be used in the recipe.

max_validation_instances (int, optional):
Maximum validation instances for the refiner.

test_refiner (StreamRefiner, optional):
Test refiner to be used in the recipe.

max_test_instances (int, optional):
Maximum test instances for the refiner.

demos_pool_size (int, optional):
Size of the demos pool. -1 for taking the whole of stream ‘demos_taken_from’.

demos_pool(List[Dict[str, Any]], optional):
a list of instances to make the demos_pool

num_demos (int, optional):
Number of demos to add to each instance, to become part of the source to be generated for this instance.

demos_taken_from (str, optional):
Specifies the stream from where the demos are taken. Default is “train”.

demos_field (str, optional):
Field name for demos. Default is “demos”. The num_demos demos selected for an instance are stored in this field of that instance.

demos_pool_field_name (str, optional):
field name to maintain the demos_pool, until sampled from, in order to make the demos. Defaults to constants.demos_pool_field.

demos_removed_from_data (bool, optional):
whether to remove the demos taken to demos_pool from the source data, Default is True

sampler (Sampler, optional):
The Sampler used to select the demonstrations when num_demos > 0.

skip_demoed_instances (bool, optional):
whether to skip pushing demos to an instance whose demos_field is already populated. Defaults to False.

steps (List[StreamingOperator], optional):
List of StreamingOperator objects to be used in the recipe.

augmentor (Augmentor) :
Augmentor to be used to pseudo randomly augment the source text

instruction_card_index (int, optional):
Index of instruction card to be used for preparing the recipe.

template_card_index (int, optional):
Index of template card to be used for preparing the recipe.

Methods:

prepare():
This overridden method is used for preparing the recipe by arranging all the steps, refiners, and renderers in a sequential manner.

Raises:

AssertionError:
If both template and template_card_index are specified at the same time.

References: templates.qa.llama_vision.multiple_choice.with_context.ai2d, templates.qa.llama_vision.with_context.chart_qa, templates.qa.llama_vision.with_context.info_vqa, templates.qa.llama_vision.with_context.doc_vqa, cards.info_vqa_lmms_eval, cards.chart_qa_lmms_eval, cards.doc_vqa.lmms_eval, formats.chat_api, cards.websrc, cards.ai2d