Hugging Face Datasets
An open source Python dataset library for local AI training and evaluation, with Hugging Face Hub access, streaming and Apache Arrow storage.
Hugging Face Datasets is a Python library for developers and researchers preparing data for AI training and evaluation. It handles local files and datasets from the Hugging Face Hub through the same data-processing tools.
- Data types and formats: Work with text, images, audio, video, PDFs, NIfTI medical images and agent traces, alongside structured files such as CSV, JSON, JSONL and Parquet.
- Large datasets: Use Apache Arrow memory-mapped storage to reduce the need to load data into RAM, or stream records without downloading the whole dataset first.
- Reusable processing: Prepare data in parallel and reuse cached processing results, with indexing, slicing and random access for inspection.
- Training integrations: Exchange data with PyTorch, TensorFlow and JAX, as well as NumPy, Pandas, Polars and Spark.
- Similarity search: Add FAISS or Elasticsearch indexes to search within datasets.
The library runs on your machine and is open source under Apache 2.0. Local file processing stays in your environment. Downloading, streaming or sharing datasets through the Hugging Face Hub uses its hosted service; Hugging Face Storage Buckets also provide hosted storage for reading and writing raw data.
Logotyper
