We present a framework for generating appropriate facial responses from a
listener in dyadic social interactions based on the speaker's words. Given an
input transcription of the speaker's words with their timestamps, our approach
autoregressively predicts a response of a listener: a sequence of listener
facial gestures, quantized using a VQ-VAE. Since gesture is a language
component, we propose treating the quantized atomic motion elements as
additional language token inputs to a transformer-based large language model.
Initializing our transformer with the weights of a language model pre-trained
only on text results in significantly higher quality listener responses than
training a transformer from scratch. We show that our generated listener motion
is fluent and reflective of language semantics through quantitative metrics and
a qualitative user study. In our evaluation, we analyze the model's ability to
utilize temporal and semantic aspects of spoken text. Project page:
this https URL

基於語言模型和量化的原子運動元素，我們提出了一種生成聽眾適當面部反應的框架，以回應演講者的言談，並展示了模型在定量指標和定性用戶研究中生成流暢且反映語義的聽眾動作的能力。

语言模型能否学会倾听？

Can Language Models Learn to Listen?

We train a MOS prediction model based on wav2vec 2.0 using the open-access
data sets BVCC and SOMOS. Our test with neural TTS data in the low-resource
language (LRL) West Frisian shows that pre-training on BVCC before fine-tuning
on SOMOS leads to the best accuracy for both fine-tuned and zero-shot
prediction. Further fine-tuning experiments show that using more than 30
percent of the total data does not lead to significant improvements. In
addition, fine-tuning with data from a single listener shows promising
system-level accuracy, supporting the viability of one-participant pilot tests.
These findings can all assist the resource-conscious development of TTS for
LRLs by progressing towards better zero-shot MOS prediction and informing the
design of listening tests, especially in early-stage evaluation.

通过对 BVCC 和 SOMOS 数据集进行 wav2vec 2.0 预训练和微调，实现了针对 LRL West Frisian 的零样本 MOS 预测。研究表明，在 30% 的数据使用量之后，使用更多的数据并没有显著性的提高性能；同时，对单个听众的数据进行微调有望提高系统级的性能和稳定性。这些发现有助于开发 LRL 语音合成系统的零样本 MOS 预测和初期评估。