Favicon of 4M

4M

An open-source multimodal AI framework for local vision tasks and image generation, with pretrained models and PyTorch CUDA support.

Screenshot of 4M website

4M is a framework for researchers and developers who want to train and run models that combine images, text and other visual information. A single model can perform vision tasks and generate images from mixed or partial inputs.

  • Vision predictions: Generate depth, surface normals, segmentation and other supported outputs from an image, or predict them from combinations of other inputs.
  • Controlled image generation: Guide image content with captions, bounding boxes, human poses, color palettes or metadata, and adjust how strongly each input influences the result.
  • Image editing: Inpaint partial images or edit depth and segmentation maps to guide composition and content.
  • Multimodal retrieval: Predict DINOv2 and ImageBind embeddings to search for images using different input types, or retrieve other visual information from an image query.
  • Training and specialization: Train multimodal models or fine-tune pretrained ones for additional tasks and input types; specialist checkpoints cover text-to-image generation and super-resolution.

Models can produce related outputs in sequence, using completed predictions to keep subsequent results consistent. The Python framework runs on your own hardware with PyTorch and CUDA. Its code is open source under Apache 2.0, and pretrained model and tokenizer checkpoints are available as safetensors files or through Hugging Face Hub.

Similar to 4M