ControlNet

ControlNet lets you guide Stable Diffusion with visual references, so a generated image can follow a sketch, a person's pose or the geometry of an existing scene. It's for artists and developers who need more control over image structure than a text prompt alone provides. The Python implementation runs on your own hardware and includes Gradio interfaces for its pretrained models.

Different controls suit different tasks:

• Canny edges and M-LSD lines guide outlines and straight structures, including buildings and interiors.
• HED boundaries retain image details for recoloring and changes in style. Scribble controls accept rough drawings or sketches derived from an image.
• Openpose supplies human pose controls, while semantic segmentation guides the arrangement of regions in a scene.
• Depth maps guide spatial structure; normal maps help preserve surface geometry and smaller shape details.

The supplied models work with Stable Diffusion 1.5, and the repository also supports Stable Diffusion 2. Guess Mode can generate images from a control map without a text prompt. Model weights and image detectors are available through Hugging Face, and a low-VRAM mode accommodates GPUs with 8 GB of memory.

ControlNet is open source under Apache 2.0. Developers can train it on their own paired images while keeping the original diffusion model's weights fixed. This approach learns the added visual control without overwriting the base model, and the project includes data annotation tools and training examples.