📄 Default¶

templates.evaluation.preference.default

MultipleChoiceTemplate(
    instruction="For any Instruction and {input_type} given to a model, assess which {output_type} written by the model aligns most closely with the given instruction (choose from {numerals}).",
    input_format="Instruction:
{instruction}
{input_type}:
{input}
Responses:
{choices}",
    target_prefix="{output_type}:
",
    target_field="output_choice",
    choices_separator="
",
    postprocessors=[
        "processors.to_string_stripped",
        "processors.first_character",
    ],
    shuffle_choices=True,
    title_fields=[
        "input_type",
        "output_type",
    ],
)
[source]

Explanation about MultipleChoiceTemplate¶

Formats the input that specifies a multiple-choice question, with a list of possible answers to choose from, and identifies the correct answer.

Args:

target_prefix (str): Optional prefix that can be added before the target label in
generated prompts or outputs.

choices_field (str): The key under which the multiple choices are stored in the
input and reference dictionaries.

target_field (str): The key under which the correct choice is stored in the
reference dictionary (can be integer index or textual label).

choices_separator (str): A string used to join formatted
choices (e.g. “, “).

source_choice_format (str): A Python format string used for displaying each choice
in the input fields (e.g. “{choice_numeral}. {choice_text}”).

target_choice_format (str): A Python format string used for displaying each choice
in the target or final output (e.g. “{choice_numeral}”).

enumerator (str): Determines how choice numerals are enumerated. Possible values
include “capitals”, “lowercase”, “numbers”, or “roman”.

shuffle_choices (bool): If True, shuffle the choices. The shuffling seed can be
set with shuffle_choices_seed.

shuffle_choices_seed (int, optional): If provided, the choices are shuffled with
this fixed integer seed for reproducibility.

sort_choices_by_length (bool): If True, sorts choices
by their length (ascending).

sort_choices_alphabetically (bool): If True, sorts choices
in alphabetical order.

reverse_choices (bool): If True, reverses the order of the choices after any
sorting has been applied. Defaults to False to preserve backward compatibility.

References: processors.to_string_stripped, processors.first_character