Favicon of Fish Diffusion

Fish Diffusion

An open-source voice model training framework for speech and singing synthesis and voice conversion, with multi-speaker and distributed training.

Screenshot of Fish Diffusion website

Fish Diffusion is a Python framework for developers and researchers training diffusion models for text-to-speech, singing voice synthesis and singing voice conversion. It works with your own datasets and provides code you can adapt for different voice generation tasks.

  • Multiple voices: Train models with support for more than one speaker.
  • Distributed training: Spread training across multiple machines and devices, with half-precision training available to reduce memory use and improve training speed.
  • Adaptable modules: Study or modify individual parts of the framework through its decoupled code structure.
  • Singing vocoder compatibility: Use the Diff Singer community vocoder at 44.1 kHz for compatible audio generation work.

This is a model training framework, so it suits people building speech or singing systems who need control over the model code and training data. Its Python environment uses Python 3.10.

Audio generation requires the FishAudio NSF-HiFiGAN vocoder. The framework's code is open source under MIT, while the downloadable FishAudio vocoder model has a separate CC BY-NC-SA 4.0 license with a noncommercial restriction. Users are responsible for obtaining permission to use their training datasets.

Similar to Fish Diffusion