Datasets and Dataloaders in JSON, CSV, and Parquet
Why? For the Fun!
Why reimplement a popular Python library in Rust? In this case, it was purely for the fun of learning. HuggingFace’s datasets library has become the go-to solution for handling large datasets for machine learning, with wide-ranging compatibility across formats and an easy-to-use API. However, the joy of learning Rust, a systems programming language known for its performance and safety, inspired this attempt to port key features of HuggingFace datasets to Rust.
This project doesn’t aim to outperform the original, nor does it seek to be a drop-in replacement. Instead, it serves as an exploratory effort into building dataloaders and handling various dataset formats using the powerful Rust language.
What?
The project implements a Rust library, hugging_datasets, which handles datasets in popular formats like JSON, CSV, and Parquet. The library provides tools for loading, splitting, and processing datasets for machine learning tasks such as classification and regression.
Key functionalities include:
- Loading datasets from JSON, CSV, or Parquet files.
- Splitting datasets into features (X) and target labels (y).
- Shuffling and splitting the datasets into train and test sets.
- The ability to use command-line arguments to specify the dataset format, input file, target column, and test ratio.
The project makes use of popular libraries like Polars for DataFrame manipulation, giving it flexibility and efficiency in handling large datasets, though its utility in a production setting is limited.
Compatibilities
The library supports the following file formats for dataset loading:
- JSON: Read and load structured datasets stored in JSON format.
- CSV: Handle flat, tabular datasets commonly used in machine learning.
- Parquet: Efficiently manage large datasets using columnar storage, which is useful for distributed systems.
- Being Blazing Fast: Felt like this had to be said since everything with Rust involved is said to be BLAZING FAST. 🔥🦀
These formats provide broad compatibility with existing dataset storage methods, ensuring that hugging_datasets can handle data from various sources.
Limitations
Despite the fun and learning benefits, there are significant limitations to this project:
- No HuggingFace Integration: While this library draws inspiration from HuggingFace’s
datasets, it does not integrate with HuggingFace’s infrastructure, APIs, or dataset hub. You must manually specify the files in the supported formats (JSON, CSV, or Parquet). - Command-line Interface: You can use the CLI to specify input/output files and the format, but there’s no integration with more advanced functionality from HuggingFace’s ecosystem, such as dataset streaming or tokenization.
- Not Optimized for Large-Scale ML: Although Rust excels in performance, this implementation is more of an academic exercise than a production-level solution. It lacks the depth and breadth of features found in HuggingFace
datasets.
Summary
This project is a fun exploration of Rust’s capabilities in handling datasets for machine learning. It demonstrates the ability to load, shuffle, split, and prepare datasets for training and testing, albeit without the depth or ecosystem of HuggingFace’s Python library. Perhaps, with more contributions or optimizations, it could become part of a larger Rust-based machine learning project like linfa. However, for now, it remains an interesting, educational project that highlights Rust’s potential for this kind of task.
With love,
— Snyata
Aylex Riom 