This project performs a Korean singing voice synthesis task using a diffusion model, specifically based on Grad-TTS. More details and demo at here
pip install -r requirements.txtWe used the Children Song Dataset, an open-source singing voice dataset comprised of 100 annotated Korean and English children songs sung by a single professional singer. We used only the Korean subset of the dataset to train the model.
You can train the model on any custom dataset of your choice, as long as it includes lyrics text, midi transcriptions, and monophonic a capella audio file triplets. These files should be titled identically, and should also be placed in specific directory locations as shown below. I used kr001a~kr047b for training, and others for test.
├── data
│ └── raw
│ ├── mid
│ ├── lyrics
│ └── wav
The directory names correspond to file extensions. We have included a sample as reference.
Once you have prepared the dataset, run
cd data_preprocess
python serialize.py
from the root directory. This will create data_preprocess/bin that contains binary files used for training. This repository already contains example binary files created from the sample in data_preprocess/raw.
ex)
python my_inference.py --song kr001a --check_points logs/0604/grad_8.ptPretrained model checkpoint for Grad-SVS can be found. here
To generate audio files with the trained model checkpoint, download the HiFi-GAN checkpoint along with its configuration file and place them in hifi-gan-mlp.
python train.py- Vocoder : HiFi-GAN.
- Architecture : Grad-TTS.
- Data preprocessing: MLP-Singer.