voicebox-pytorch

Voicebox - Pytorch

在Pytorch中实现<a href="https://arxiv.org/abs/2306.15687">Voicebox</a>，来自MetaAI的新一代文本转语音模型。<a href="https://about.fb.com/news/2023/06/introducing-voicebox-ai-for-speech-generation/">新闻稿</a>

在这项工作中，我们将使用旋转嵌入。作者似乎没有意识到ALiBi不能简单地用于双向模型。

该论文还解决了时间嵌入错误地受到相对距离影响的问题（他们沿音频令牌的帧维度连接时间嵌入）。这个库将使用自适应归一化，正如在<a href="https://arxiv.org/abs/2211.07292">Paella</a>中成功应用的那样。

感谢

感谢<a href="https://translated.com"><img style="vertical-align: middle;" src="https://yellow-cdn.veclightyear.com/35dd4d3f/75d53761-8018-4c3b-8ca1-53570fe306cd.png" height="20px" alt="Translated"><img></a>授予我<a href="https://imminent.translated.com/research-grants-ceremony-innovations-in-language-technology">Imminent Grant</a>，以推进开源码文本转语音解决方案的状态。本项目在这项资助下启动并将完成。
感谢<a href="https://stability.ai/">StabilityAI</a>的慷慨赞助，以及我的其他赞助商，使我能够有独立性开发开源人工智能。
感谢<a href="https://github.com/b-chiang">Bryan Chiang</a>的持续代码审查，分享他在TTS方面的专业知识，并指引我到<a href="https://github.com/atong01/conditional-flow-matching">一个开源实现</a>的条件流匹配中。
感谢<a href="https://github.com/manmay-nakhashi">Manmay</a>帮助这个库以对齐代码开始。
感谢<a href="https://github.com/chenht2010">@chenht2010</a>发现旋转位置的一个bug，并验证库中的代码可以收敛。
感谢<a href="https://github.com/lucasnewman">Lucas Newman</a>（再次）提交了所有用于Spear-TTS条件Voicebox训练的训练代码的pull request！
感谢<a href="https://github.com/lucasnewman">Lucas Newman</a>展示了整个系统在Spear-TTS条件下运行正常。训练收敛效果比<a href="https://github.com/lucidrains/soundstorm-pytorch">Soundstorm</a>还要好。

安装

$ pip install voicebox-pytorch

使用

使用<a href="https://github.com/lucidrains/spear-tts-pytorch">SpearTTS</a>中的TextToSemantic模块进行训练和采样

import torch

from voicebox_pytorch import (
    VoiceBox,
    EncodecVoco,
    ConditionalFlowMatcherWrapper,
    HubertWithKmeans,
    TextToSemantic
)

# https://github.com/facebookresearch/fairseq/tree/main/examples/hubert

wav2vec = HubertWithKmeans(
    checkpoint_path = '/path/to/hubert/checkpoint.pt',
    kmeans_path = '/path/to/hubert/kmeans.bin'
)

text_to_semantic = TextToSemantic(
    wav2vec = wav2vec,
    dim = 512,
    source_depth = 1,
    target_depth = 1,
    use_openai_tokenizer = True
)

text_to_semantic.load('/path/to/trained/spear-tts/model.pt')

model = VoiceBox(
    dim = 512,
    audio_enc_dec = EncodecVoco(),
    num_cond_tokens = 500,
    depth = 2,
    dim_head = 64,
    heads = 16
)

cfm_wrapper = ConditionalFlowMatcherWrapper(
    voicebox = model,
    text_to_semantic = text_to_semantic
)

# 模拟数据

audio = torch.randn(2, 12000)

# 训练

loss = cfm_wrapper(audio)
loss.backward()

# 经过大量训练之后

texts = [
    '西班牙的雨水主要落在平原上',
    '她在海边卖海贝壳'
]

cond = torch.randn(2, 12000)
sampled = cfm_wrapper.sample(cond = cond, texts = texts) # (2, 1, <音频长度>)

对于无条件训练，VoiceBox上的condition_on_text必须设置为False

import torch
from voicebox_pytorch import (
    VoiceBox,
    ConditionalFlowMatcherWrapper
)

model = VoiceBox(
    dim = 512,
    num_cond_tokens = 500,
    depth = 2,
    dim_head = 64,
    heads = 16,
    condition_on_text = False
)

cfm_wrapper = ConditionalFlowMatcherWrapper(
    voicebox = model
)

# 模拟数据

x = torch.randn(2, 1024, 512)

# 训练

loss = cfm_wrapper(x)

loss.backward()

# 经过大量训练之后

cond = torch.randn(2, 1024, 512)

sampled = cfm_wrapper.sample(cond = cond) # (2, 1024, 512)

待办事项

引用

@article{Le2023VoiceboxTM,
    title   = {Voicebox: Text-Guided Multilingual Universal Speech Generation at Scale},
    author  = {Matt Le and Apoorv Vyas and Bowen Shi and Brian Karrer and Leda Sari and Rashel Moritz and Mary Williamson and Vimal Manohar and Yossi Adi and Jay Mahadeokar and Wei-Ning Hsu},
    journal = {ArXiv},
    year    = {2023},
    volume  = {abs/2306.15687},
    url     = {https://api.semanticscholar.org/CorpusID:259275061}
}

@inproceedings{dao2022flashattention,
    title   = {Flash{A}ttention: Fast and Memory-Efficient Exact Attention with {IO}-Awareness},
    author  = {Dao, Tri and Fu, Daniel Y. and Ermon, Stefano and Rudra, Atri and R{\'e}, Christopher},
    booktitle = {Advances in Neural Information Processing Systems},
    year    = {2022}
}

@misc{torchdiffeq,
    author  = {Chen, Ricky T. Q.},
    title   = {torchdiffeq},
    year    = {2018},
    url     = {https://github.com/rtqichen/torchdiffeq},
}

@inproceedings{lienen2022torchode,
    title     = {torchode: A Parallel {ODE} Solver for PyTorch},
    author    = {Marten Lienen and Stephan G{\"u}nnemann},
    booktitle = {The Symbiosis of Deep Learning and Differential Equations II, NeurIPS},
    year      = {2022},
    url       = {https://openreview.net/forum?id=uiKVKTiUYB0}
}

@article{siuzdak2023vocos,
    title   = {Vocos: Closing the gap between time-domain and Fourier-based neural vocoders for high-quality audio synthesis},
    author  = {Siuzdak, Hubert},
    journal = {arXiv preprint arXiv:2306.00814},
    year    = {2023}
}

@misc{darcet2023vision,
    title   = {Vision Transformers Need Registers},
    author  = {Timothée Darcet and Maxime Oquab and Julien Mairal and Piotr Bojanowski},
    year    = {2023},
    eprint  = {2309.16588},
    archivePrefix = {arXiv},
    primaryClass = {cs.CV}
}

@inproceedings{Dehghani2023ScalingVT,
    title   = {Scaling Vision Transformers to 22 Billion Parameters},
    author  = {Mostafa Dehghani and Josip Djolonga and Basil Mustafa and Piotr Padlewski and Jonathan Heek and Justin Gilmer and Andreas Steiner and Mathilde Caron and Robert Geirhos and Ibrahim M. Alabdulmohsin and Rodolphe Jenatton and Lucas Beyer and Michael Tschannen and Anurag Arnab and Xiao Wang and Carlos Riquelme and Matthias Minderer and Joan Puigcerver and Utku Evci and Manoj Kumar and Sjoerd van Steenkiste and Gamaleldin F. Elsayed and Aravindh Mahendran and Fisher Yu and Avital Oliver and Fantine Huot and Jasmijn Bastings and Mark Collier and Alexey A. Gritsenko and Vighnesh Birodkar and Cristina Nader Vasconcelos and Yi Tay and Thomas Mensink and Alexander Kolesnikov and Filip Paveti'c and Dustin Tran and Thomas Kipf and Mario Luvci'c and Xiaohua Zhai and Daniel Keysers and Jeremiah Harmsen and Neil Houlsby},
    booktitle = {International Conference on Machine Learning},
    year    = {2023},
    url     = {https://api.semanticscholar.org/CorpusID:256808367}
}

@inproceedings{Katsch2023GateLoopFD,
    title   = {GateLoop: Fully Data-Controlled Linear Recurrence for Sequence Modeling},
    author  = {Tobias Katsch},
    year    = {2023},
    url     = {https://api.semanticscholar.org/CorpusID:265018962}
}