Neural speech codec has recently gained widespread attention in generative
speech modeling domains, like voice conversion, text-to-speech synthesis, etc.
However, ensuring high-fidelity audio reconstruction of speech codecs under
high compression rates remains an open and challenging issue. In this paper, we
propose PromptCodec, a novel end-to-end neural speech codec model using
disentangled representation learning based feature-aware prompt encoders. By
incorporating additional feature representations from prompt encoders,
PromptCodec can distribute the speech information requiring processing and
enhance its capabilities. Moreover, a simple yet effective adaptive feature
weighted fusion approach is introduced to integrate features of different
encoders. Meanwhile, we propose a novel disentangled representation learning
strategy based on cosine distance to optimize PromptCodec's encoders to ensure
their efficiency, thereby further improving the performance of PromptCodec.
Experiments on LibriTTS demonstrate that our proposed PromptCodec consistently
outperforms state-of-the-art neural speech codec models under all different
bitrate conditions while achieving impressive performance with low bitrates.

利用解耦表示学习为基准的特征感知提示编码器，我们提出了 PromptCodec，这是一种创新的端到端神经语音编解码模型，用于高压缩比下确保音频重建的高保真度。PromptCodec 通过整合提示编码器的额外特征表示，可以分配需要处理的语音信息并提高其能力。此外，我们引入了一种简单但有效的自适应特征加权融合方法，用于整合不同编码器的特征，并基于余弦距离提出了一种新颖的解耦表示学习策略以优化 PromptCodec 的编码器，从而进一步提高其性能。在 LibriTTS 上的实验证明，我们提出的 PromptCodec 在不同比特率条件下始终优于最先进的神经语音编解码模型，并在低比特率下取得了令人印象深刻的性能。

PromptCodec：基于分离表示学习的自适应特征感知提示编码的高保真神经语音编解码器

PromptCodec: High-Fidelity Neural Speech Codec using Disentangled  Representation Learning based Adaptive Feature-aware Prompt Encoders

This paper presents FunCodec, a fundamental neural speech codec toolkit,
which is an extension of the open-source speech processing toolkit FunASR.
FunCodec provides reproducible training recipes and inference scripts for the
latest neural speech codec models, such as SoundStream and Encodec. Thanks to
the unified design with FunASR, FunCodec can be easily integrated into
downstream tasks, such as speech recognition. Along with FunCodec, pre-trained
models are also provided, which can be used for academic or generalized
purposes. Based on the toolkit, we further propose the frequency-domain codec
models, FreqCodec, which can achieve comparable speech quality with much lower
computation and parameter complexity. Experimental results show that, under the
same compression ratio, FunCodec can achieve better reconstruction quality
compared with other toolkits and released models. We also demonstrate that the
pre-trained models are suitable for downstream tasks, including automatic
speech recognition and personalized text-to-speech synthesis. This toolkit is
publicly available at this https URL

FunCodec 是一个基于开源语音处理工具 FunASR 的基础神经语音编解码器工具包，提供训练可重现、推理脚本用于最新的神经语音编解码器模型，如 SoundStream 和 Encodec，并且可以轻松集成到下游任务中，如语音识别。其中，还提供预训练模型，可用于学术或一般用途。此工具包还进一步提出了频域编解码器模型 FreqCodec，与其他工具包和发布模型相比，在相同的压缩比下，FunCodec 可以实现更好的重构质量，并且证明预训练模型适用于包括自动语音识别和个性化文本转语音合成在内的下游任务。

FunCodec: 一个基础、可重现和可集成的神经语音编解码开源工具包

FunCodec: A Fundamental, Reproducible and Integrable Open-source Toolkit  for Neural Speech Codec

Since facial actions such as lip movements contain significant information
about speech content, it is not surprising that audio-visual speech enhancement
methods are more accurate than their audio-only counterparts. Yet,
state-of-the-art approaches still struggle to generate clean, realistic speech
without noise artifacts and unnatural distortions in challenging acoustic
environments. In this paper, we propose a novel audio-visual speech enhancement
framework for high-fidelity telecommunications in AR/VR. Our approach leverages
audio-visual speech cues to generate the codes of a neural speech codec,
enabling efficient synthesis of clean, realistic speech from noisy signals.
Given the importance of speaker-specific cues in speech, we focus on developing
personalized models that work well for individual speakers. We demonstrate the
efficacy of our approach on a new audio-visual speech dataset collected in an
unconstrained, large vocabulary setting, as well as existing audio-visual
datasets, outperforming speech enhancement baselines on both quantitative
metrics and human evaluation studies. Please see the supplemental video for
qualitative results at
this https URL

本文提出了一种新的音频 - 视觉语音增强框架，利用个人化模型和神经语音编解码器从嘈杂的信号中高效合成真实干净的语音，以提高增强幅度和视角方面的质量。