unitxt.stream module

class unitxt.stream.DynamicStream(gen_kwargs: Dict[str, Any] = {}, caching: bool = False, copying: bool = False)

Bases: Stream

peek()

set_copying(copying: bool)

take(n)

exception unitxt.stream.EmptyStreamError

Bases: FaultyStreamError

Raised when a stream is unexpectedly empty.

exception unitxt.stream.FaultyStreamError

Bases: Exception

Base class for all stream-related exceptions.

class unitxt.stream.GeneratorStream(gen_kwargs: Dict[str, Any] = {}, caching: bool = False, copying: bool = False)

Bases: Stream

A class for handling streaming data in a customizable way.

This class provides methods for generating, caching, and manipulating streaming data.

generator

A generator function for streaming data. :no-index:

Type:

function

gen_kwargs

A dictionary of keyword arguments for the generator function. :no-index:

Type:

dict, optional

caching

Whether the data is cached or not. :no-index:

Type:

bool

peek()

set_copying(copying: bool)

take(n)

class unitxt.stream.ListStream(copying: bool = False)

Bases: Stream

peek()

set_copying(copying: bool)

take(n) → Generator

exception unitxt.stream.MissingStreamError

Bases: FaultyStreamError

Raised when a required stream is missing.

class unitxt.stream.MultiStream(data=None)

Bases: dict

A class for handling multiple streams of data in a dictionary-like format.

This class extends dict and its values should be instances of the Stream class.

data

A dictionary of Stream objects.

Type:

dict

classmethod from_generators(generators: Dict[str, ReusableGenerator], caching=False, copying=False)

Creates a MultiStream from a dictionary of ReusableGenerators.

Parameters:

• generators (Dict[str, ReusableGenerator]) – A dictionary of ReusableGenerators.

• caching (bool, optional) – Whether the data should be cached or not. Defaults to False.

• copying (bool, optional) – Whether the data should be copied or not. Defaults to False.

Returns:

A MultiStream object.

Return type:

MultiStream

classmethod from_iterables(iterables: Dict[str, Iterable], caching=False, copying=False)

Creates a MultiStream from a dictionary of iterables.

Parameters:

• iterables (Dict[str, Iterable]) – A dictionary of iterables.

• caching (bool, optional) – Whether the data should be cached or not. Defaults to False.

• copying (bool, optional) – Whether the data should be copied or not. Defaults to False.

Returns:

A MultiStream object.

Return type:

MultiStream

get_generator(key) → Generator

Gets a generator for a specified key.

Parameters:

key (str) – The key for the generator.

Yields:

object – The next value in the stream.

set_caching(caching: bool)

set_copying(copying: bool)

to_dataset(disable_cache=True, cache_dir=None) → DatasetDict

to_iterable_dataset() → IterableDatasetDict

class unitxt.stream.Stream

Bases: Dataclass

abstract peek()

abstract set_copying(copying: bool)

abstract take(n)

unitxt.stream.eager_failed()