📄 Default¶

templates.evaluation.preference.default

type: MultipleChoiceTemplate
instruction: For any Instruction and {input_type} given to a model, assess which {output_type} written by the model aligns most closely with the given instruction (choose from {numerals}).
input_format: "Instruction:\n{instruction}\n{input_type}:\n{input}\nResponses:\n{choices}"
target_prefix: "{output_type}:\n"
target_field: output_choice
choices_separator: "\n"
postprocessors: 
  - processors.to_string_stripped
  - processors.first_character
shuffle_choices: True
title_fields: 
  - input_type
  - output_type
[source]

Explanation about MultipleChoiceTemplate¶

Formats the input (that specifies the question), the multiple choices to select the answer from, and specifies the field with the correct answer.

References: processors.to_string_stripped, processors.first_character