FastSpeech is a Transformer- based feedforward network used to generate TTS mel spectrograms in parallel. Compared with the autoregressive Transformer TTS , the FastSpeech model can generate mel spectrograms 270 times faster and end-to-end speech synthesis 38 times faster.
Project realization
docker cp LJSpeech-1.1.tar.bz2 torch_na:/workspace/FastSpeech/data
docker cp /home/elena/tts/waveglow_256channels_ljs_v2.pt torch_na:/workspace/FastSpeech/waveglow/pretrained_modelRename the downloaded pre-trained model to
waveglow_256channels.pt
Unzip the file as the current folder
unzip alignments.zip Then run preprocess.py
python preprocess.py After processing the data, start training
python train.py
(Training for nearly a week) As shown in the picture after training
To verify after training, first modify the hyperparameter --step in eval.py to the number after the checkpoint in the model_new folder you just trained, as shown in the figure
In my case, I changed the default 0 to 768000, as follows
if __name__ == "__main__":
# Test
WaveGlow = utils.get_WaveGlow()
parser = argparse.ArgumentParser()
parser.add_argument('--step', type=int, default=768000) #把默认0，改为768000，其他的不变
parser.add_argument("--alpha", type=float, default=1.0)
args = parser.parse_args()and then run
python eval.py(If an error occurs after running, please refer to Q&A2)
The results are shown as shown in the figure
Two results are generated, one is through mel_spce and the other is through waveglow.
The effect generated by waveglow is better, and the noise of mel_space is relatively large!
Detailed code explanation
preprocess.py -> Preprocess the LJSpeech data set
def preprocess_ljspeech(filename):
# LJSpeech 数据集作为输入路径
in_dir = filename
# mel 谱图输出路径为 ./mels ，若路径不存在则创建路径
out_dir = hp.mel_ground_truth
if not os.path.exists(out_dir):
os.makedirs(out_dir, exist_ok=True)
# 执行语音波形-mel谱图转换，并保存mel谱图，得到LJSpeech数据集语音文本列表
metadata = ljspeech.build_from_path(in_dir, out_dir)
# 将得到的语音文本列表写入磁盘
write_metadata(metadata, out_dir)
# 移动语音文本列表文件
shutil.move(os.path.join(hp.mel_ground_truth, "train.txt"),
os.path.join("data", "train.txt"))
hparams.py model related parameters
# Mel
num_mels = 80
text_cleaners = ['english_cleaners']
# FastSpeech
vocab_size = 300
max_seq_len = 3000
encoder_dim = 256 #模型编码维度
encoder_n_layer = 4 #模型编码层数
encoder_head = 2 #模型头
encoder_conv1d_filter_size = 1024 #模型输出大小
decoder_dim = 256 #模型解码维度
decoder_n_layer = 4
decoder_head = 2
decoder_conv1d_filter_size = 1024
fft_conv1d_kernel = (9, 1)
fft_conv1d_padding = (4, 0)
duration_predictor_filter_size = 256
duration_predictor_kernel_size = 3
dropout = 0.1
# Train
checkpoint_path = "./model_new" #训练模型保存路径
logger_path = "./logger" #训练日志保存路径
mel_ground_truth = "./mels" #
alignment_path = "./alignments"
batch_size = 32
epochs = 2000
n_warm_up_step = 4000
learning_rate = 1e-3
weight_decay = 1e-6
grad_clip_thresh = 1.0
decay_step = [500000, 1000000, 2000000]
save_step = 3000
log_step = 5
clear_Time = 20
batch_expand_size = 32
Q&A
1. Why is the generated map map (mel) an npy file?
2.ModuleNotFoundError: No module named 'numba.decorators'
When running the validation model, a model error occurs because of the wrong library version.
Uninstall numba and then install numba-0.48.0
pip install numba==0.48.0references
【1】GitHub - xcmyz/FastSpeech: The Implementation of FastSpeech based on pytorch.