基于联合对比学习策略的单样本语音转换模型

One-Shot Voice Conversion Model Based on Combined Contrastive Learning

摘要

针对单样本 (one-shot) 语音转换任务中目标说话人内容保持与说话人特征迁移难以兼顾的问题，本文在变分推理语音合成 (Variational Inference Text-to-Speech, VITS) 框架的基础上，提出了基于联合对比学习策略的单样本语音转换模型。该模型在内容与说话人两个特征空间同时引入对比约束。针对内容特征提取，将 WavLM (Waveform Language Model) 第 23 层的输出经 K 最近邻 (K-Nearest Neighbors, KNN) 匹配后，与 WavLM 第 24 层的输出结合，构成样本对，以实现音素级对齐和语义一致性。针对说话人特征提取，通过时序扰动生成对比样本，并采用监督对比损失，提取稳定且与内容无关的说话人嵌入。生成器部分通过结合条件变分自编码器 (Conditional Variational Auto Encoder, CVAE)与对抗损失进行优化训练。实验结果表明，该模型在 VCTK (Voice Cloning Tool Kit) 语料库上获得的客观与主观评价指标均优于最新基线模型，验证了所提模型在语音自然度、内容保持以及说话人相似性方面的有效提升。

Abstract

To address the challenge of balancing content preservation of the target speaker and speaker characteristic transfer in one-shot voice conversion tasks, this paper proposes a one-shot voice conversion model based on a joint contrastive learning strategy, built upon the Variational Inference Text-to-Speech (VITS) framework. The model introduces contrastive constraints in both the content and speaker feature spaces simultaneously. For content feature extraction, the output from the 23rd layer of Waveform Language Model（WavLM） is combined with the output from the 24th layer of WavLM after K-Nearest Neighbors (KNN) matching to form sample pairs, achieving phoneme-level alignment and semantic consistency. For speaker feature extraction, contrastive samples are generated via temporal perturbation, and supervised contrastive loss is adopted to extract speaker embeddings that are stable and content-independent. The generator is trained with the combination of a Conditional Variational AutoEncoder (CVAE) and adversarial loss optimization. Experimental results on the Voice Cloning Tool Kit (VCTK) corpus show that the proposed model outperforms state-of-the-art baseline models in both objective and subjective evaluation metrics, verifying its effectiveness in improving speech naturalness, content preservation, and speaker similarity.

关键词

单样本 / 语音转换 / 对比学习 / K 最近邻算法 / 时序扰乱

Key words

one-shot / voice conversion / contrastive learning / KNN regression / temporal perturbation

参考文献

[1] WU D Y, CHEN Y H, LEE H Y. VQVC+: One-shot voice conversion by vector quantization and u-net architecture[C]//Proceedings of the Annual Conference of the International Speech Communication Association. Shanghai: ISCA, 2020:4691-4695.

[2] YANG S C, TANTRAWENITH M, ZHUANG H, et al. Speech representation disentanglement with adversarial mutual information learning for one-shot voice conversion[C]//Proceedings of the Annual Conference of the International Speech Communication Association. Incheon: ISCA, 2022:2553-2557.

[3] LI Y A, HAN C, MESGARANI N. Styletts-VC: One-shot voice conversion by knowledge transfer from style-based tts models[C]//IEEE Spoken Language Technology Workshop. Doha: IEEE, 2023:920-927.

[4] LU H, WANG D, WU X, et al. Disentangled speech representation learning for one-shot cross-lingual voice conversion using β-vae[C]//IEEE Spoken Language Technology Workshop. Doha: IEEE, 2023:814-821.

[5] PARK H J, YANG S W, KIM J S, et al. Triaan-VC: Triple adaptive attention normalization for any-to-any voice conversion[C]//IEEE International Conference on Acoustics, Speech and Signal Processing. Rhodes Island: IEEE, 2023:1-5.

[6] LIN J, LIN Y Y, CHIEN C M, et al. S2VC: A framework for any-to-any voice conversion with self-supervised pretrained representations[C]//Proceedings of the Annual Conference of the International Speech Communication Association. Brno: ISCA, 2021:836-840.

[7] LIN Y Y, CHIEN C M, LIN J H, et al. FragmentVC: Any-to-any voice conversion by end-to-end extracting and fusing fine-grained voice fragments with attention[C]//IEEE International Conference on Acoustics, Speech and Signal Processing. Toronto: IEEE, 2021:5939-5943.

[8] QIAN K, ZHANG Y, GAO H, et al. Contentvec: An improved self-supervised speech representation by disentangling speakers[C]//Proceedings of the 39th International Conference on Machine Learning. Baltimore: PMLR, 2022:18003-18017.

[9] HUANG H, WANG L, YANG J, et al. W2VC: WavLM representation based one-shot voice conversion with gradient reversal distillation and CTC supervision[J].EURASIP Journal on Audio, Speech, and Music Processing, 2023, 2023(1):45.

[10] GUO Y, LI Z, LI J, et al. Vec2wav 2.0: Advancing voice conversion via discrete token vocoders[EB/OL]. (2024-09-03). https://arxiv.org/abs/2409.01995.

[11] HUANG L, YUAN T, LIANG Y, et al. Limi-VC: A light weight voice conversion model with mutual information disentanglement[C]//IEEE International Conference on Acoustics, Speech and Signal Processing. Rhodes Island: IEEE, 2023:1-5.

[12] KIM J, KONG J, SON J. Conditional variational autoencoder with adversarial learning for end-to-end text-to-speech[C]//International Conference on Machine Learning. Vienna: PMLR, 2021:5530-5540.

[13] LI J, TU W, XIAO L. Freevc: Towards high-quality text-free one-shot voice conversion[C]//IEEE International Conference on Acoustics, Speech and Signal Processing. Rhodes Island: IEEE, 2023:1-5.

[14] GUO H, LIU C, ISHI C T, et al. QuickVC: A lightweight vits-based any-to-many voice conversion model using istft for faster conversion[C]//IEEE Automatic Speech Recognition and Understanding Workshop. Taipei, China: IEEE, 2023:1-7.

[15] VAN NIEKERK B, CARBONNEAU M A, ZAÏDI J, et al. A comparison of discrete and soft speech units for improved voice conversion[C]//IEEE International Conference on Acoustics, Speech and Signal Processing. Singapore: IEEE, 2022:6562-6566.

[16] CAO D, ZHANG Z, ZHANG J. NeuralVC: Any-to-any voice conversion using neural networks decoder for real-time voice conversion[J].IEEE Signal Processing Letters, 2024, 31:2070-2074.

[17] LI T, LIU Y, HU C, et al. CVC: Contrastive learning for non-parallel voice conversion[C]//Proceedings of the Annual Conference of the International Speech Communication Association. Brno: ISCA, 2021:1324-1328.

[18] TANG H, ZHANG X, WANG J, et al. VQ-CL: Learning disentangled speech representations with contrastive learning and vector quantization[C]//IEEE International Conference on Acoustics, Speech and Signal Processing. Rhodes Island: IEEE, 2023:1-5.

[19] TANG H, ZHANG X, WANG J, et al. AVQVC: One-shot voice conversion by vector quantization with applying contrastive learning[C]//IEEE International Conference on Acoustics, Speech and Signal Processing. Singapore: IEEE, 2022:4613-4617.

[20] PRIHASTO B, LIN Y X, LE P T, et al. CNEG-VC: contrastive learning using hard negative example in non-parallel voice conversion[C]//IEEE International Conference on Acoustics, Speech and Signal Processing. Rhodes Island: IEEE, 2023:1-5.

[21] DENG Y, ZHANG X, WANG J, et al. CLN-VC: Text-free voice conversion based on fine-grained style control and contrastive learning with negative samples augmentation[C]//IEEE International Conference on Parallel and Distributed Processing with Applications, Big Data and Cloud Computing, Sustainable Computing and Communications, Social Computing and Networking. Wuhan: IEEE, 2023:1143-1148.

[22] XUE Y, CHEN N, LUO Y, et al. Clessr-VC: Contrastive learning enhanced self-supervised representations for one-shot voice conversion[J].Speech Communication, 2024, 165:1-8.

[23] AHANGARAN M, DAWALATABAD N, KARJADI C, et al. Obfuscation via pitch‐shifting for balancing privacy and diagnostic utility in voice‐ based cognitive assessment[J].Alzheimer's & Dementia, 2025, 21(3):e70032.

[24] QIAN K, JIN Z, HASEGAWA-JOHNSON M, et al. F0-consistent many-to-many non-parallel voice conversion via conditional autoencoder[C]//IEEE International Conference on Acoustics, Speech and Signal Processing. Barcelona: IEEE, 2020:6284-6288.

[25] YAMAGISHI J, VEAUX C, MACDONALD K. CSTR VCTK corpus: English multi-speaker corpus for cstr voice cloning toolkit (version 0.92)[R]. Edinburgh: University of Edinburgh,2019.

[26] CHEN S, WANG C, CHEN Z, et al. WavLM: Large-scale self-supervised pre-training for full stack speech processing[J].IEEE Journal of Selected Topics in Signal Processing, 2022, 16(6):1505-1518.

[27] BAAS M, VAN NIEKERK B, KAMPER H. Voice conversion with just nearest neighbors[C]//Proceedings of the Annual Conference of the International Speech Communication Association. Dublin: ISCA, 2023:2053-2057.

[28] PARULEKAR A, COLLINS L, SHANMUGAM K, et al. Infonce loss provably learns cluster-preserving representations[C]//The Thirty Sixth Annual Conference on Learning Theory. Bangalore: PMLR, 2023:1914-1961.

[29] HWANG Y, CHO H, YANG H, et al. Mel-spectrogram augmentation for sequence to sequence voice conversion[EB/OL]. (2020-01-06). https://arxiv.org/abs/2001.01401.

[30] KHOSLA P, TETERWAK P, WANG C, et al. Supervised contrastive learning[C]//Advances in Neural Information Processing Systems. Virtual: Curran Associates Inc, 2020:18661-18673.

[31] KONG J, KIM J, BAE J. Hifi-gan: Generative adversarial networks for efficient and high fidelity speech synthesis[C]//Advances in Neural Information Processing Systems. Virtual: Curran Associates Inc, 2020:17022-17033.

[32] LOSHCHILOV I, HUTTER F. Decoupled weight decay regularization[C]//International Conference on Learning Representations. New Orleans: OpenReview, 2019:1-8.

[33] HALDER R K, UDDIN M N, UDDIN M A, et al. Enhancing k-nearest neighbor algorithm: A comprehensive review and performance analysis of modifications[J]. Journal of Big Data, 2024, 11: 113.

[34] KIM J, KIM J H, CHOI Y, et al. AdaptVC: High quality voice conversion with adaptive learning[C]//IEEE International Conference on Acoustics, Speech and Signal Processing. Hyderabad: IEEE, 2025:1-5.

[35] MITTAG G, MÖLLER S. Deep learning based assessment of synthetic speech naturalness[C]//Proceedings of the Annual Conference of the International Speech Communication Association. Shanghai: ISCA, 2020:1748-1752.

[36] MAATEN L, HINTON G. Visualizing data using t-SNE[J].Journal of Machine Learning Research, 2008, 9:2579-2605.

[37] WATANABE S, HORI T, KARITA S, et al. ESPnet: End-to-end speech processing toolkit[C]//Proceedings of the Annual Conference of the International Speech Communication Association. Hyderabad: ISCA, 2018:2207-2211.

基金资助

国家自然科学基金面上项目(61771196)