DataDreamer
A Python library for synthetic data generation and LLM training that supports local and API-based models, with caching and resumable workflows.
DataDreamer is a Python library for researchers and developers who need to create training datasets and adapt LLMs to specific tasks. It connects data preparation and model training in reproducible workflows.
- Prompting and datasets: Combine multiple prompting steps to generate examples for a new task, expand an existing dataset, clean its contents, or create few-shot examples.
- Model training: Fine-tune, instruction-tune, distill, or align models with human preferences using existing or generated data, including Hugging Face models such as T5.
- Repeatable experiments: Caching and resumability let workflows reuse completed work after an interruption, while shareable workflows help researchers reproduce experiments.
- Training resources: LoRA and quantization can reduce resource demands, and the library supports parallel workflow steps and model execution or training across multiple GPUs.
- Publishing results: Send datasets and trained models to Hugging Face Hub, with generated data cards, model cards, metadata, and citation lists.
DataDreamer is open source under the MIT license. It supports models running on your own hardware as well as API-based LLMs. Prompting through OpenAI models such as GPT-4 sends requests to OpenAI's cloud service; publishing to Hugging Face Hub uploads the selected datasets or models there.
Logotyper
