Favicon of ControlNet

ControlNet

An open-source Stable Diffusion component that guides local image generation with sketches, edges, poses and depth maps under Apache 2.0.

ControlNet adds visual guidance to Stable Diffusion for artists and developers who need generated images to follow a reference's layout or shape. It lets a sketch, pose or scene map guide the result alongside a text prompt.

  • Outlines and sketches: Use Canny edges, M-LSD straight lines or scribbles to guide object contours and structures such as buildings and interiors.
  • Recoloring and style changes: HED boundaries retain details from a reference image while guiding changes to its appearance.
  • Human poses: Openpose detects a person's pose in an input image and uses it to guide the generated figure.
  • Scene geometry: Semantic segmentation guides the arrangement of scene regions, while depth and normal maps guide spatial structure and surface details.

The Python implementation runs on your own hardware and includes Gradio interfaces for the supplied pretrained models, which work with Stable Diffusion 1.5. Model weights and the associated image detectors are available from Hugging Face. ControlNet is open source under Apache 2.0.

Visual guidance doesn't guarantee an exact copy of the reference; generation can still alter image details.

Similar to ControlNet