unitxt.loaders module¶
This section describes unitxt loaders.
Loaders: Generators of Unitxt Multistreams from existing date sources¶
Unitxt is all about readily preparing of any given data source for feeding into any given language model, and then, post-processing the model’s output, preparing it for any given evaluator.
Through that journey, the data advances in the form of Unitxt Multistream, undergoing a sequential application of various off-the-shelf operators (i.e., picked from Unitxt catalog), or operators easily implemented by inheriting. The journey starts by a Unitxt Loader bearing a Multistream from the given datasource. A loader, therefore, is the first item on any Unitxt Recipe.
Unitxt catalog contains several loaders for the most popular datasource formats. All these loaders inherit from Loader, and hence, implementing a loader to expand over a new type of datasource is straightforward.
- Available Loaders Overview:
LoadHF
- Loads data from HuggingFace Datasets.LoadCSV
- Imports data from CSV (Comma-Separated Values) files.LoadFromKaggle
- Retrieves datasets from the Kaggle community site.LoadFromIBMCloud
- Fetches datasets hosted on IBM Cloud.LoadFromSklearn
- Loads datasets available through the sklearn library.MultipleSourceLoader
- Combines data from multiple different sources.LoadFromDictionary
- Loads data from a user-defined Python dictionary.LoadFromHFSpace
- Downloads and loads data from HuggingFace Spaces.
- class unitxt.loaders.LazyLoader(data_classification_policy: List[str] = None, _requirements_list: List[str] | Dict[str, str] = [], requirements: List[str] | Dict[str, str] = [], caching: bool = None, loader_limit: int = None, streaming: bool = False, num_proc: int = None, split: str | NoneType = None)[source]¶
Bases:
Loader
- class unitxt.loaders.LoadCSV(data_classification_policy: List[str] = None, _requirements_list: List[str] | Dict[str, str] = [], requirements: List[str] | Dict[str, str] = [], caching: bool = None, loader_limit: int | NoneType = None, streaming: bool = True, num_proc: int = None, split: str | NoneType = None, files: Dict[str, str] = __required__, chunksize: int = 1000, sep: str = ',', compression: str | NoneType = None, lines: bool | NoneType = None, file_type: Literal['csv', 'json'] = 'csv')[source]¶
Bases:
LazyLoader
Loads data from CSV files.
Supports streaming and can handle large files by loading them in chunks.
- Parameters:
files (Dict[str, str]) – A dictionary mapping names to file paths.
chunksize – Size of the chunks to load at a time.
loader_limit – Optional integer to specify a limit on the number of records to load.
streaming – Bool indicating if streaming should be used.
sep – String specifying the separator used in the CSV files.
Example
Loading csv
load_csv = LoadCSV(files={'train': 'path/to/train.csv'}, chunksize=100)
- class unitxt.loaders.LoadFromAPI(data_classification_policy: List[str] = None, _requirements_list: List[str] | Dict[str, str] = [], requirements: List[str] | Dict[str, str] = [], caching: bool = None, loader_limit: int | NoneType = None, streaming: bool = False, num_proc: int = None, urls: Dict[str, str] = __required__, chunksize: int = 100000, api_key_env_var: str = 'SQL_API_KEY', headers: Dict[str, Any] | NoneType = None, data_field: str = 'data', method: str = 'GET')[source]¶
Bases:
Loader
Loads data from from API.
This loader is designed to fetch data from an API endpoint, handling authentication through an API key. It supports customizable chunk sizes and limits for data retrieval.
- Parameters:
urls (Dict[str, str]) – A dictionary mapping split names to their respective API URLs.
chunksize (int, optional) – The size of data chunks to fetch in each request. Defaults to 100,000.
loader_limit (int, optional) – Limits the number of records to load. Applied per split. Defaults to None.
streaming (bool, optional) – Determines if data should be streamed. Defaults to False.
api_key_env_var (str, optional) – The name of the environment variable holding the API key. Defaults to “SQL_API_KEY”.
headers (Dict[str, Any], optional) – Additional headers to include in API requests. Defaults to None.
data_field (str, optional) – The name of the field in the API response that contains the data. Defaults to “data”.
method (str, optional) – The HTTP method to use for API requests. Defaults to “GET”.
- class unitxt.loaders.LoadFromDictionary(data_classification_policy: List[str] = None, _requirements_list: List[str] | Dict[str, str] = [], requirements: List[str] | Dict[str, str] = [], caching: bool = None, loader_limit: int = None, streaming: bool = False, num_proc: int = None, data: Dict[str, List[Dict[str, Any]]] = __required__)[source]¶
Bases:
Loader
Allows loading data from a dictionary of constants.
The loader can be used, for example, when debugging or working with small datasets.
- Parameters:
data (Dict[str, List[Dict[str, Any]]]) – a dictionary of constants from which the data will be loaded
Example
Loading dictionary
data = { "train": [{"input": "SomeInput1", "output": "SomeResult1"}, {"input": "SomeInput2", "output": "SomeResult2"}], "test": [{"input": "SomeInput3", "output": "SomeResult3"}, {"input": "SomeInput4", "output": "SomeResult4"}] } loader = LoadFromDictionary(data=data)
- class unitxt.loaders.LoadFromHFSpace(data_classification_policy: List[str] = None, _requirements_list: List[str] | Dict[str, str] = [], requirements: List[str] | Dict[str, str] = [], caching: bool = None, loader_limit: int = None, streaming: bool = True, num_proc: int | NoneType = None, split: str | NoneType = None, path: str | NoneType = None, name: str | NoneType = None, data_dir: str | NoneType = None, data_files: str | Sequence[str] | Mapping[str, str | Sequence[str]] = __required__, revision: str | NoneType = None, filtering_lambda: str | NoneType = None, splits: List[str] | NoneType = None, space_name: str = __required__, use_token: bool | NoneType = None, token_env: str | NoneType = None, requirements_list: List[str] = ['huggingface_hub'])[source]¶
Bases:
LoadHF
Used to load data from HuggingFace Spaces.
Loaders firstly tries to download all files specified in the ‘data_files’ parameter from the given space and then reads them as a HuggingFace Dataset.
- Parameters:
space_name (str) – Name of the HuggingFace Space to be accessed.
data_files (str | Sequence[str] | Mapping[str, str | Sequence[str]]) – Relative paths to files within a given repository. If given as a mapping, paths should be values, while keys should represent the type of respective files (training, testing etc.).
path (str, optional) – Absolute path to a directory where data should be downloaded.
revision (str, optional) – ID of a Git branch or commit to be used. By default, it is set to None, thus data is downloaded from the main branch of the accessed repository.
use_token (bool, optional) – Whether a token is used for authentication when accessing the HuggingFace Space. If necessary, the token is read from the HuggingFace config folder.
token_env (str, optional) – Key of an env variable which value will be used for authentication when accessing the HuggingFace Space - if necessary.
Example
Loading from a HuggingFace Space
loader = LoadFromHFSpace( space_name="lmsys/mt-bench", data_files={ "train": [ "data/mt_bench/model_answer/gpt-3.5-turbo.jsonl", "data/mt_bench/model_answer/gpt-4.jsonl", ], "test": "data/mt_bench/model_answer/tulu-30b.jsonl", }, )
- requirements_list: List[str] = ['huggingface_hub']¶
- class unitxt.loaders.LoadFromIBMCloud(data_classification_policy: List[str] = ['proprietary'], _requirements_list: List[str] = ['ibm-cos-sdk'], requirements: List[str] | Dict[str, str] = [], caching: bool = True, loader_limit: int = None, streaming: bool = False, num_proc: int = None, endpoint_url_env: str = __required__, aws_access_key_id_env: str = __required__, aws_secret_access_key_env: str = __required__, bucket_name: str = __required__, data_dir: str = None, data_files: Sequence[str] | Mapping[str, str | Sequence[str]] = __required__, data_field: str = None)[source]¶
Bases:
Loader
Loads data from IBM Cloud Object Storage.
Does not support streaming and requires AWS-style access keys. data_dir Can be either: 1. a list of file names, the split of each file is determined by the file name pattern 2. Mapping: split -> file_name, e.g. {“test” : “test.json”, “train”: “train.json”} 3. Mapping: split -> file_names, e.g. {“test” : [“test1.json”, “test2.json”], “train”: [“train.json”]}
- Parameters:
endpoint_url_env – Environment variable name for the IBM Cloud endpoint URL.
aws_access_key_id_env – Environment variable name for the AWS access key ID.
aws_secret_access_key_env – Environment variable name for the AWS secret access key.
bucket_name – Name of the S3 bucket from which to load data.
data_dir – Optional directory path within the bucket.
data_files – Union type allowing either a list of file names or a mapping of splits to file names.
data_field – The dataset key for nested JSON file, i.e. when multiple datasets are nested in the same file
caching (bool) – indicating if caching is enabled to avoid re-downloading data.
Example
Loading from IBM Cloud
load_ibm_cloud = LoadFromIBMCloud( endpoint_url_env='IBM_CLOUD_ENDPOINT', aws_access_key_id_env='IBM_AWS_ACCESS_KEY_ID', aws_secret_access_key_env='IBM_AWS_SECRET_ACCESS_KEY', bucket_name='my-bucket' ) multi_stream = load_ibm_cloud.process()
- data_classification_policy: List[str] = ['proprietary']¶
- class unitxt.loaders.LoadFromKaggle(data_classification_policy: List[str] = ['public'], _requirements_list: List[str] = ['opendatasets'], requirements: List[str] | Dict[str, str] = [], caching: bool = None, loader_limit: int = None, streaming: bool = False, num_proc: int = None, url: str = __required__)[source]¶
Bases:
Loader
Loads datasets from Kaggle.
Requires Kaggle API credentials and does not support streaming.
- Parameters:
url – URL to the Kaggle dataset.
Example
Loading from kaggle
load_kaggle = LoadFromKaggle(url='kaggle.com/dataset/example')
- data_classification_policy: List[str] = ['public']¶
- class unitxt.loaders.LoadFromSklearn(data_classification_policy: List[str] = ['public'], _requirements_list: List[str] = ['scikit-learn', 'pandas'], requirements: List[str] | Dict[str, str] = [], caching: bool = None, loader_limit: int = None, streaming: bool = False, num_proc: int = None, split: str | NoneType = None, dataset_name: str = __required__, splits: List[str] = ['train', 'test'])[source]¶
Bases:
LazyLoader
Loads datasets from the sklearn library.
This loader does not support streaming and is intended for use with sklearn’s dataset fetch functions.
- Parameters:
dataset_name – The name of the sklearn dataset to fetch.
splits – A list of data splits to load, e.g., [‘train’, ‘test’].
Example
Loading form sklearn
load_sklearn = LoadFromSklearn(dataset_name='iris', splits=['train', 'test'])
- data_classification_policy: List[str] = ['public']¶
- splits: List[str] = ['train', 'test']¶
- class unitxt.loaders.LoadHF(data_classification_policy: List[str] = None, _requirements_list: List[str] | Dict[str, str] = [], requirements: List[str] | Dict[str, str] = [], caching: bool = None, loader_limit: int = None, streaming: bool = None, num_proc: int | NoneType = None, split: str | NoneType = None, path: str = __required__, name: str | NoneType = None, data_dir: str | NoneType = None, data_files: str | Sequence[str] | Mapping[str, str | Sequence[str]] | NoneType = None, revision: str | NoneType = None, filtering_lambda: str | NoneType = None, splits: List[str] | NoneType = None)[source]¶
Bases:
LazyLoader
Loads datasets from the HuggingFace Hub.
It supports loading with or without streaming, and it can filter datasets upon loading.
- Parameters:
path – The path or identifier of the dataset on the HuggingFace Hub.
name – An optional dataset name.
data_dir – Optional directory to store downloaded data.
split – Optional specification of which split to load.
data_files – Optional specification of particular data files to load.
revision – Optional. The revision of the dataset. Often the commit id. Use in case you want to set the dataset version.
streaming (bool) – indicating if streaming should be used.
filtering_lambda (str, optional) – A lambda function for filtering the data after loading.
num_proc (int, optional) – Specifies the number of processes to use for parallel dataset loading.
Example
Loading glue’s mrpc dataset
load_hf = LoadHF(path='glue', name='mrpc')
- class unitxt.loaders.Loader(data_classification_policy: List[str] = None, _requirements_list: List[str] | Dict[str, str] = [], requirements: List[str] | Dict[str, str] = [], caching: bool = None, loader_limit: int = None, streaming: bool = False, num_proc: int = None)[source]¶
Bases:
SourceOperator
A base class for all loaders.
A loader is the first component in the Unitxt Recipe, responsible for loading data from various sources and preparing it as a MultiStream for processing. The loader_limit is an optional parameter used to control the maximum number of instances to load from the data source. It is applied for each split separately. It is usually provided to the loader via the recipe (see standard.py) The loader can use this value to limit the amount of data downloaded from the source to reduce loading time. However, this may not always be possible, so the loader may ignore this. In any case, the recipe, will limit the number of instances in the returned stream, after load is complete.
- Parameters:
loader_limit – Optional integer to specify a limit on the number of records to load.
streaming – Bool indicating if streaming should be used.
num_proc – Optional integer to specify the number of processes to use for parallel dataset loading. Adjust the value according to the number of CPU cores available and the specific needs of your processing task.
- class unitxt.loaders.MultipleSourceLoader(data_classification_policy: List[str] = None, _requirements_list: List[str] | Dict[str, str] = [], requirements: List[str] | Dict[str, str] = [], caching: bool = None, loader_limit: int = None, streaming: bool = False, num_proc: int = None, sources: List[unitxt.loaders.Loader] = __required__)[source]¶
Bases:
Loader
Allows loading data from multiple sources, potentially mixing different types of loaders.
- Parameters:
sources – A list of loaders that will be combined to form a unified dataset.
Examples
Loading the train split from a HuggingFace Hub and the test set from a local file:
MultipleSourceLoader(sources = [ LoadHF(path="public/data",split="train"), LoadCSV({"test": "mytest.csv"}) ])
Loading a test set combined from two files
MultipleSourceLoader(sources = [ LoadCSV({"test": "mytest1.csv"}, LoadCSV({"test": "mytest2.csv"}) ])