JoyCaption

JoyCaption generates image descriptions for people preparing datasets to train or fine-tune diffusion models. It can caption images on your own GPU, reducing the work of writing training descriptions by hand.

• Image coverage: Caption photography, digital art, anime and furry artwork, with support for both SFW and NSFW content.
• Caption control: Choose detailed descriptions and guide their length, tone and focus, including lighting, camera angle and composition, or exclude personal attributes and sexual details.
• Workflow integrations: Use the ComfyUI node for image workflows or serve the model through vLLM's OpenAI-compatible API.
• Fine-tuning: Adapt the model to your own image data with the supplied training scripts.

The model is free to download with open weights, and the repository's code uses Apache 2.0. Local inference runs on your hardware; the Hugging Face Spaces demo runs online. At native precision, the model needs about 17GB of VRAM, with 24GB or more recommended for comfortable use. The ComfyUI node supports 8-bit and 4-bit quantization for lower memory use.

JoyCaption is designed for captioning and doesn't reliably follow arbitrary instructions. It can confuse left and right, misread image text and struggle with multiple subjects. Some NSFW questions can still trigger refusals.