CleanRL

CleanRL is a Python deep reinforcement learning library for researchers and developers who want to study or modify an algorithm without tracing it through a large modular codebase. Each algorithm variant lives in one standalone file. Experiments can run locally, while Docker and AWS integration support cloud execution through AWS Batch.

The single-file approach makes the training logic accessible when you're debugging an experiment or prototyping a feature that other libraries don't support. It also means duplicated code across variants. CleanRL isn't designed as a modular library to import into an application.

Its algorithms include PPO, DQN, C51, DDPG, TD3, SAC, and PPG. Variants cover Atari games and continuous-action tasks, with implementations that use PyTorch and others that use JAX. PPO variants also include recurrent policies and multi-GPU training. The focus is online reinforcement learning, where agents learn through interaction with an environment, rather than offline learning from a fixed dataset.

For evaluating experiments, CleanRL provides TensorBoard logging, seeded runs for local reproducibility, and gameplay video capture. Weights and Biases handles experiment tracking. Its benchmark reports expose recorded training results, GPU utilization, and gameplay videos, and the related Open RL Benchmark includes experiments from Stable-baselines3, openai/baselines, and jaxrl. Install the source checkout and run its training scripts directly. The documented environment requires Python 3.7.1 or later but below 3.11; optional environment extras have their own dependencies. Code is MIT licensed.