unitxt.struct_data_operators module

This section describes unitxt operators for tabular data.

These operators are specialized in handling tabular data. Input table format is assumed as: {

“header”: [“col1”, “col2”], “rows”: [[“row11”, “row12”], [“row21”, “row22”], [“row31”, “row32”]]

}

class unitxt.struct_data_operators.ListToKeyValPairs(apply_to_streams: ~typing.List[str] = None, dont_apply_to_streams: ~typing.List[str] = None, fields: ~typing.List[str], to_field: str, use_query: bool = False)

Bases: StreamInstanceOperator

Maps list of keys and values into key:value pairs.

Sample input in expected format: {“keys”: [“name”, “age”, “sex”], “values”: [“Alex”, 31, “M”]} Sample output: {“name”: “Alex”, “age”: 31, “sex”: “M”}

class unitxt.struct_data_operators.SerializeKeyValPairs(apply_to_streams: List[str] = None, dont_apply_to_streams: List[str] = None, field: str | None = None, to_field: str | None = None, field_to_field: List[List[str]] | Dict[str, str] | None = None, process_every_value: bool = False, use_query: bool = False, get_default: Any = None, not_exist_ok: bool = False)

Bases: FieldOperator

Serializes key, value pairs into a flat sequence.

Sample input in expected format: {“name”: “Alex”, “age”: 31, “sex”: “M”} Sample output: name is Alex, age is 31, sex is M

class unitxt.struct_data_operators.SerializeTable(apply_to_streams: List[str] = None, dont_apply_to_streams: List[str] = None, field: str | None = None, to_field: str | None = None, field_to_field: List[List[str]] | Dict[str, str] | None = None, process_every_value: bool = False, use_query: bool = False, get_default: Any = None, not_exist_ok: bool = False)

Bases: ABC, FieldOperator

TableSerializer converts a given table into a flat sequence with special symbols.

Output format varies depending on the chosen serializer. This abstract class defines structure of a typical table serializer that any concrete implementation should follow.

class unitxt.struct_data_operators.SerializeTableAsIndexedRowMajor(apply_to_streams: List[str] = None, dont_apply_to_streams: List[str] = None, field: str | None = None, to_field: str | None = None, field_to_field: List[List[str]] | Dict[str, str] | None = None, process_every_value: bool = False, use_query: bool = False, get_default: Any = None, not_exist_ok: bool = False)

Bases: SerializeTable

Indexed Row Major Table Serializer.

Commonly used row major serialization format. Format: col : col1 | col2 | col 3 row 1 : val1 | val2 | val3 | val4 row 2 : val1 | …

class unitxt.struct_data_operators.SerializeTableAsMarkdown(apply_to_streams: List[str] = None, dont_apply_to_streams: List[str] = None, field: str | None = None, to_field: str | None = None, field_to_field: List[List[str]] | Dict[str, str] | None = None, process_every_value: bool = False, use_query: bool = False, get_default: Any = None, not_exist_ok: bool = False)

Bases: SerializeTable

Markdown Table Serializer.

Markdown table format is used in GitHub code primarily. Format: |col1|col2|col3| |---|—|---| |A|4|1| |I|2|1| …

class unitxt.struct_data_operators.SerializeTableRowAsList(apply_to_streams: ~typing.List[str] = None, dont_apply_to_streams: ~typing.List[str] = None, fields: str, to_field: str, max_cell_length: int | None = None)

Bases: StreamInstanceOperator

Serializes a table row as list.

Parameters:

  • fields (str) –

  • to_field (str) –

  • max_cell_length (int) –

class unitxt.struct_data_operators.SerializeTableRowAsText(apply_to_streams: ~typing.List[str] = None, dont_apply_to_streams: ~typing.List[str] = None, fields: str, to_field: str, max_cell_length: int | None = None)

Bases: StreamInstanceOperator

Serializes a table row as text.

Parameters:

  • fields (str) –

  • to_field (str) –

  • max_cell_length (int) –

class unitxt.struct_data_operators.SerializeTriples(apply_to_streams: List[str] = None, dont_apply_to_streams: List[str] = None, field: str | None = None, to_field: str | None = None, field_to_field: List[List[str]] | Dict[str, str] | None = None, process_every_value: bool = False, use_query: bool = False, get_default: Any = None, not_exist_ok: bool = False)

Bases: FieldOperator

Serializes triples into a flat sequence.

Sample input in expected format: [[ “First Clearing”, “LOCATION”, “On NYS 52 1 Mi. Youngsville” ], [ “On NYS 52 1 Mi. Youngsville”, “CITY_OR_TOWN”, “Callicoon, New York”]]

Sample output: First Clearing : LOCATION : On NYS 52 1 Mi. Youngsville | On NYS 52 1 Mi. Youngsville : CITY_OR_TOWN : Callicoon, New York

class unitxt.struct_data_operators.TruncateTableCells(apply_to_streams: List[str] = None, dont_apply_to_streams: List[str] = None, max_length: int = 15, table: str = None, text_output: str | None = None, use_query: bool = False)

Bases: StreamInstanceOperator

Limit the maximum length of cell values in a table to reduce the overall length.

Parameters:

  • max_length (int) –

  • answer (For tasks that produce a cell value as) –

  • replicated (truncating a cell value should be) –

  • implementation. (with truncating the corresponding answer as well. This has been addressed in the) –

class unitxt.struct_data_operators.TruncateTableRows(apply_to_streams: List[str] = None, dont_apply_to_streams: List[str] = None, field: str | None = None, to_field: str | None = None, field_to_field: List[List[str]] | Dict[str, str] | None = None, process_every_value: bool = False, use_query: bool = False, get_default: Any = None, not_exist_ok: bool = False, rows_to_keep: int = 10)

Bases: FieldOperator

Limits table rows to specified limit by removing excess rows via random selection.

Parameters:

rows_to_keep (int) –

unitxt.struct_data_operators.truncate_cell(cell_value, max_len)