📄 Response Generation¶

This is a task corresponding to the response generation step of RAG pipeline. It assumes the input for is a set of questions and already retrieved contexts (documents or passsages). The model response answer is evaluated against a set of reference_answers and/or using referenceless metrics such as the faithfullness of the model answer to the provided context.

This task is similar to ‘task.qa.with_context’ , but supports multiple contexts and is focused only on text.

For details of RAG see: https://www.unitxt.ai/en/latest/docs/rag_support.html.

tasks.rag.response_generation

Task(
    input_fields={
        "contexts": "List[str]",
        "contexts_ids": "Union[List[int], List[str]]",
        "question": "str",
    },
    reference_fields={
        "reference_answers": "List[str]",
    },
    metrics=[
        "metrics.rag.response_generation.correctness.token_overlap",
        "metrics.rag.response_generation.faithfullness.token_overlap",
        "metrics.rag.response_generation.correctness.bert_score.deberta_large_mnli",
    ],
    augmentable_inputs=[
        "contexts",
        "question",
    ],
    prediction_type="str",
    defaults={
        "contexts_ids": [],
        "reference_answers": [],
    },
)
[source]

Explanation about Task¶

Task packs the different instance fields into dictionaries by their roles in the task.

Args:

input_fields (Union[Dict[str, str], List[str]]):
Dictionary with string names of instance input fields and types of respective values. In case a list is passed, each type will be assumed to be Any.

reference_fields (Union[Dict[str, str], List[str]]):
Dictionary with string names of instance output fields and types of respective values. In case a list is passed, each type will be assumed to be Any.

metrics (List[str]):
List of names of metrics to be used in the task.

prediction_type (Optional[str]):
Need to be consistent with all used metrics. Defaults to None, which means that it will be set to Any.

defaults (Optional[Dict[str, Any]]):
An optional dictionary with default values for chosen input/output keys. Needs to be consistent with names and types provided in ‘input_fields’ and/or ‘output_fields’ arguments. Will not overwrite values if already provided in a given instance.

The output instance contains three fields:

“input_fields” whose value is a sub-dictionary of the input instance, consisting of all the fields listed in Arg ‘input_fields’.

“reference_fields” – for the fields listed in Arg “reference_fields”.

“metrics” – to contain the value of Arg ‘metrics’

References: metrics.rag.response_generation.correctness.bert_score.deberta_large_mnli, metrics.rag.response_generation.faithfullness.token_overlap, metrics.rag.response_generation.correctness.token_overlap