Dask-ML
An open-source Python machine learning library that uses Dask to handle datasets larger than RAM and parallelize work with Scikit-Learn and XGBoost.
Dask-ML is a Python machine learning library for people whose datasets exceed RAM or whose training and evaluation jobs take too long. It uses Dask to distribute work across a cluster of machines, while keeping interfaces familiar to users of Scikit-Learn, NumPy, and pandas. It's open source under the BSD-3-Clause license.
The library addresses two different bottlenecks. When a model fits in memory but takes too long to train, predict, or evaluate, Dask-ML can parallelize the work. Its tools include hyperparameter optimization, and Dask's joblib integration can distribute Scikit-Learn workloads while retaining familiar data structures.
For datasets too large to load into NumPy or pandas, Dask-ML provides estimators that work with Dask's data collections. These include preprocessing and ensemble methods, so users can prepare data and build models without requiring the entire dataset to fit in RAM.
Its appeal is continuity with an existing Python workflow. Users can scale parts of their work without adopting an entirely different machine learning interface. For XGBoost, Dask-ML relies on XGBoost's own distributed capabilities: Dask handles data preparation and passes the data to XGBoost running alongside it.
Logotyper
