In this paper, we study the application of Test-Time Training (TTT) as a
solution to handling distribution shifts in speech applications. In particular,
we introduce distribution-shifts to the test datasets of standard
speech-classification tasks -- for example, speaker-identification and
emotion-detection -- and explore how Test-Time Training (TTT) can help adjust
to the distribution-shift. In our experiments that include distribution shifts
due to background noise and natural variations in speech such as gender and
age, we identify some key-challenges with TTT including sensitivity to
optimization hyperparameters (e.g., number of optimization steps and subset of
parameters chosen for TTT) and scalability (e.g., as each example gets its own
set of parameters, TTT is not scalable). Finally, we propose using BitFit -- a
parameter-efficient fine-tuning algorithm proposed for text applications that
only considers the bias parameters for fine-tuning -- as a solution to the
aforementioned challenges and demonstrate that it is consistently more stable
than fine-tuning all the parameters of the model.

本文研究了测试时间训练（TTT）在处理语音应用中的分布偏移问题的应用。我们引入分布偏移到标准语音分类任务的测试数据集中，并探索了测试时间训练（TTT）如何帮助调整到分布偏移。在我们的实验中，包括背景噪声和语音中的性别和年龄等自然变化引起的分布偏移，我们确定了 TTT 面临的一些关键挑战，包括对优化超参数（例如，优化步数和选择 TTT 的参数子集）的敏感性以及可扩展性（例如，由于每个示例都有自己的参数集，TTT 不具有可扩展性）。最后，我们提出使用 BitFit 作为解决上述挑战的参数高效微调算法，BitFit 仅考虑偏差参数进行微调，并证明它相对于微调模型的所有参数更加稳定。

测试时训练语音

Test-Time Training for Speech

Many speech applications require understanding aspects beyond the words being
spoken, such as recognizing emotion, detecting whether the speaker is wearing a
mask, or distinguishing real from synthetic speech. In this work, we introduce
a new state-of-the-art paralinguistic representation derived from large-scale,
fully self-supervised training of a 600M+ parameter Conformer-based
architecture. We benchmark on a diverse set of speech tasks and demonstrate
that simple linear classifiers trained on top of our time-averaged
representation outperform nearly all previous results, in some cases by large
margins. Our analyses of context-window size demonstrate that, surprisingly, 2
second context-windows achieve 96\% the performance of the Conformers that use
the full long-term context on 7 out of 9 tasks. Furthermore, while the best
per-task representations are extracted internally in the network, stable
performance across several layers allows a single universal representation to
reach near optimal performance on all tasks.

本研究介绍了一种基于 Conformer 架构的自我监督训练的优秀的全语境表示法，应用于多元语音任务，分析了上下文窗口的大小并证明了小窗口仍能取得接近最优表现。

使用自监督转换器的通用语音语音语言表示

Universal Paralinguistic Speech Representations Using Self-Supervised  Conformers

Deep generative models have achieved great success in unsupervised learning
with the ability to capture complex nonlinear relationships between latent
generating factors and observations. Among them, a factorized hierarchical
variational autoencoder (FHVAE) is a variational inference-based model that
formulates a hierarchical generative process for sequential data. Specifically,
an FHVAE model can learn disentangled and interpretable representations, which
have been proven useful for numerous speech applications, such as speaker
verification, robust speech recognition, and voice conversion. However, as we
will elaborate in this paper, the training algorithm proposed in the original
paper is not scalable to datasets of thousands of hours, which makes this model
less applicable on a larger scale. After identifying limitations in terms of
runtime, memory, and hyperparameter optimization, we propose a hierarchical
sampling training algorithm to address all three issues. Our proposed method is
evaluated comprehensively on a wide variety of datasets, ranging from 3 to
1,000 hours and involving different types of generating factors, such as
recording conditions and noise types. In addition, we also present a new
visualization method for qualitatively evaluating the performance with respect
to the interpretability and disentanglement. Models trained with our proposed
algorithm demonstrate the desired characteristics on all the datasets.

本文介绍一种基于变分推断的分层生成过程的因式分解分层变分自编码器（FHVAE）模型，该模型在许多语音应用方面已被证明可行，但其原始论文中提出的训练算法不适用于规模更大的数据集。本文提出了一种分层采样训练算法来解决这一问题，并对不同类型的数据集进行了全面评估，证明了我们提出的算法对所有数据集都表现出所需的特性。