📄 Arena Hard Hf Space Processing Steps¶
Note
ID: operators.arena_hard_hf_space_processing_steps | Type: SequentialOperator
{
"__type__": "sequential_operator",
"steps": [
{
"__type__": "rename_fields",
"apply_to_streams": [
"questions"
],
"field_to_field": {
"cluster": "group"
}
},
{
"__type__": "copy",
"apply_to_streams": [
"questions"
],
"field_to_field": {
"turns/0/content": "model_input"
}
},
{
"__type__": "copy",
"apply_to_streams": [
"model_answer"
],
"field_to_field": {
"choices/0/turns/0/content": "model_output",
"choices/0/turns/0/token_len": "model_output_token_len"
}
},
{
"__type__": "apply",
"_argv": [
"model_id"
],
"apply_to_streams": [
"model_answer"
],
"function": "str.lower",
"to_field": "model_id"
},
{
"__type__": "copy",
"apply_to_streams": [
"judgment"
],
"field_to_field": {
"games/0/judgment": "judge_output_model_1_ordered_first",
"games/0/score": "score_model_1_ordered_first",
"games/0/user_prompt": "judge_input_model_1_ordered_first",
"games/1/judgment": "judge_output_model_2_ordered_first",
"games/1/score": "score_model_2_ordered_first",
"games/1/user_prompt": "judge_input_model_2_ordered_first"
}
},
{
"__type__": "rename_fields",
"apply_to_streams": [
"judgment"
],
"field_to_field": {
"judge": "judge_model_id",
"model": "model_2"
}
},
{
"__type__": "set",
"apply_to_streams": [
"judgment"
],
"fields": {
"model_1": "gpt-4-0314"
}
},
{
"__type__": "apply",
"_argv": [
"judge_input_model_1_ordered_first"
],
"apply_to_streams": [
"judgment"
],
"function": "str",
"to_field": "judge_input_model_1_ordered_first"
},
{
"__type__": "apply",
"_argv": [
"judge_input_model_2_ordered_first"
],
"apply_to_streams": [
"judgment"
],
"function": "str",
"to_field": "judge_input_model_2_ordered_first"
},
{
"__type__": "apply",
"_argv": [
"model_1"
],
"apply_to_streams": [
"judgment"
],
"function": "str.lower",
"to_field": "model_1"
},
{
"__type__": "apply",
"_argv": [
"model_2"
],
"apply_to_streams": [
"judgment"
],
"function": "str.lower",
"to_field": "model_2"
},
{
"__type__": "filter_by_condition",
"apply_to_streams": [
"judgment"
],
"condition": "in",
"values": {
"score_model_1_ordered_first": [
"A=B",
"A>B",
"A>>B",
"B>A",
"B>>A"
],
"score_model_2_ordered_first": [
"A=B",
"A>B",
"A>>B",
"B>A",
"B>>A"
]
}
},
{
"__type__": "join_streams",
"how": "inner",
"left_stream": "questions",
"new_stream_name": "merged_stream",
"on": [
"question_id"
],
"right_stream": "judgment"
},
{
"__type__": "rename_fields",
"apply_to_streams": [
"model_answer"
],
"field_to_field": {
"model_id": "model_1",
"model_output": "model_1_output"
}
},
{
"__type__": "join_streams",
"how": "inner",
"left_stream": "merged_stream",
"new_stream_name": "merged_stream",
"on": [
"question_id",
"model_1"
],
"right_stream": "model_answer"
},
{
"__type__": "rename_fields",
"apply_to_streams": [
"model_answer"
],
"field_to_field": {
"model_1": "model_2",
"model_1_output": "model_2_output"
}
},
{
"__type__": "join_streams",
"how": "inner",
"left_stream": "merged_stream",
"new_stream_name": "merged_stream",
"on": [
"question_id",
"model_2"
],
"right_stream": "model_answer"
},
{
"__type__": "delete_splits",
"splits": [
"questions",
"model_answer",
"judgment"
]
},
{
"__type__": "rename_splits",
"mapper": {
"merged_stream": "test"
}
},
{
"__type__": "select_fields",
"fields": [
"question_id",
"category",
"model_input",
"model_1",
"model_2",
"judge_model_id",
"model_1_output",
"model_2_output",
"score_model_1_ordered_first",
"score_model_2_ordered_first",
"judge_input_model_1_ordered_first",
"judge_input_model_2_ordered_first",
"judge_output_model_1_ordered_first",
"judge_output_model_2_ordered_first"
]
}
]
}
Explanation about SequentialOperator¶
A class representing a sequential operator in the streaming system.
A sequential operator is a type of MultiStreamOperator that applies a sequence of other operators to a MultiStream. It maintains a list of StreamingOperator`s and applies them in order to the `MultiStream.
Explanation about FilterByCondition¶
Filters a stream, yielding only instances in which the values in required fields follow the required condition operator.
Raises an error if a required field name is missing from the input instance.
- Args:
values (Dict[str, Any]): Field names and respective Values that instances must match according the condition, to be included in the output. condition: the name of the desired condition operator between the specified (sub) field’s value and the provided constant value. Supported conditions are (“gt”, “ge”, “lt”, “le”, “ne”, “eq”, “in”,”not in”) error_on_filtered_all (bool, optional): If True, raises an error if all instances are filtered out. Defaults to True.
- Examples:
FilterByCondition(values = {“a”:4}, condition = “gt”) will yield only instances where field “a” contains a value > 4 FilterByCondition(values = {“a”:4}, condition = “le”) will yield only instances where “a”<=4 FilterByCondition(values = {“a”:[4,8]}, condition = “in”) will yield only instances where “a” is 4 or 8 FilterByCondition(values = {“a”:[4,8]}, condition = “not in”) will yield only instances where “a” different from 4 or 8 FilterByCondition(values = {“a/b”:[4,8]}, condition = “not in”) will yield only instances where “a” is
a dict in which key “b” is mapped to a value that is neither 4 nor 8
- FilterByCondition(values = {“a[2]”:4}, condition = “le”) will yield only instances where “a” is a list whose 3-rd
element is <= 4
Explanation about JoinStreams¶
Join multiple streams into a single stream.
- Args:
left_stream (str): The stream that will be considered the “left” in the join operations. right_stream (str): The stream that will be considered the “right” in the join operations. how (Literal[“left”, “right”, “inner”, “outer”, “cross”]): The type of join to be performed. on (Optional[List[str]]): Column names to join on. These must be found in both streams. left_on (Optional[List[str]]): Column names to join on in the left stream. right_on (Optional[List[str]]): Column names to join on in the right streasm. new_stream_name (str): The name of the new stream resulting from the merge.
- Examples:
JoinStreams(left_stream = “questions”, right_stream = “answers”, how=”inner”, on=”question_id”, new_stream_name=”question_with_answers” ) Join the ‘question’ and ‘answer’ stream based on the ‘question_id’ field using inner join, resulting with a new stream named “question_with_answers”. JoinStreams(left_stream = “questions”, right_stream = “answers”, how=”inner”, on_left=”question_id”, on_right=”question” new_stream_name=”question_with_answers” ) Join the ‘question’ and ‘answer’ stream based on the ‘question_id’ field in the left stream and the ‘question’ field in the right stream, using inner join, resulting with a new stream named “question_with_answers”. This is suitable when the fields have different labels across the streams.
Explanation about Copy¶
Copies values from specified fields to specified fields.
- Args (of parent class):
field_to_field (Union[List[List], Dict[str, str]]): A list of lists, where each sublist contains the source field and the destination field, or a dictionary mapping source fields to destination fields.
- Examples:
An input instance {“a”: 2, “b”: 3}, when processed by Copy(field_to_field={“a”: “b”} would yield {“a”: 2, “b”: 2}, and when processed by Copy(field_to_field={“a”: “c”} would yield {“a”: 2, “b”: 3, “c”: 2}
with field names containing / , we can also copy inside the field: Copy(field=”a/0”,to_field=”a”) would process instance {“a”: [1, 3]} into {“a”: 1}
Explanation about DeleteSplits¶
Operator which delete splits in stream.
- Attributes:
splits (List[str]): The splits to delete from the stream.
Explanation about Set¶
Adds specified fields to each instance in a given stream or all streams (default) If fields exist, updates them.
- Args:
- fields (Dict[str, object]): The fields to add to each instance.
Use ‘/’ to access inner fields
use_deepcopy (bool) : Deep copy the input value to avoid later modifications
- Examples:
# Add a ‘classes’ field with a value of a list “positive” and “negative” to all streams Set(fields={“classes”: [“positive”,”negatives”]})
# Add a ‘start’ field under the ‘span’ field with a value of 0 to all streams Set(fields={“span/start”: 0}
# Add a ‘classes’ field with a value of a list “positive” and “negative” to ‘train’ stream Set(fields={“classes”: [“positive”,”negatives”], apply_to_stream=[“train”]})
# Add a ‘classes’ field on a given list, prevent modification of original list # from changing the instance. Set(fields={“classes”: alist}), use_deepcopy=True) # if now alist is modified, still the instances remain intact.
Explanation about RenameFields¶
Renames fields.
Move value from one field to another, potentially, if field name contains a /, from one branch into another. Remove the from field, potentially part of it in case of / in from_field.
- Examples:
RenameFields(field_to_field={“b”: “c”}) will change inputs [{“a”: 1, “b”: 2}, {“a”: 2, “b”: 3}] to [{“a”: 1, “c”: 2}, {“a”: 2, “c”: 3}]
RenameFields(field_to_field={“b”: “c/d”}) will change inputs [{“a”: 1, “b”: 2}, {“a”: 2, “b”: 3}] to [{“a”: 1, “c”: {“d”: 2}}, {“a”: 2, “c”: {“d”: 3}}]
RenameFields(field_to_field={“b”: “b/d”}) will change inputs [{“a”: 1, “b”: 2}, {“a”: 2, “b”: 3}] to [{“a”: 1, “b”: {“d”: 2}}, {“a”: 2, “b”: {“d”: 3}}]
RenameFields(field_to_field={“b/c/e”: “b/d”}) will change inputs [{“a”: 1, “b”: {“c”: {“e”: 2, “f”: 20}}}] to [{“a”: 1, “b”: {“c”: {“f”: 20}, “d”: 2}}]
Explanation about Apply¶
A class used to apply a python function and store the result in a field.
- Args:
function (str): name of function. to_field (str): the field to store the result additional arguments are field names passed to the function
Examples: Store in field “b” the uppercase string of the value in field “a” Apply(“a”, function=str.upper, to_field=”b”)
Dump the json representation of field “t” and store back in the same field. Apply(“t”, function=json.dumps, to_field=”t”)
Set the time in a field ‘b’. Apply(function=time.time, to_field=”b”)
Explanation about SelectFields¶
Keep only specified fields from each instance in a stream.
- Args:
fields (List[str]): The fields to keep from each instance.
Read more about catalog usage here.