Hivemind

Hivemind is a Python library for researchers and teams that want to train a shared neural network across computers owned by universities, companies or volunteers. It uses PyTorch and connects participants over the internet, so contributors can pool their machines for a training job without a central master node.

Its training design accommodates peers that respond slowly or become unavailable. Forward and backward passes can still succeed when some nodes don't respond, and workers can average model updates without waiting for the entire network to synchronize. That makes it relevant to collaborative training where participants can't guarantee the availability of every computer.

For models too large to keep together on one machine, Hivemind supports a decentralized Mixture-of-Experts approach that distributes parts of neural network layers across participants. A distributed hash table connects the peers. The library also supports optional blockwise 8-bit compression through bitsandbytes for data transfers between workers.

Hivemind is open source under the MIT license. Linux is its primary development and testing platform; macOS has partial support, and a Docker image is available. The software runs on participating computers, while collaborative training exchanges updates across the network. Its examples include a starter kit for jointly training an ALBERT Transformer masked language model.

Windows support is experimental and requires WSL. Collaborative training shares model updates with the chosen peers; installing the library locally does not make a public training network private.