Evaluating the theory of mind (ToM) capabilities of language models (LMs) has
recently received much attention. However, many existing benchmarks rely on
synthetic data which risks misaligning the resulting experiments with human
behavior. We introduce the first ToM dataset based on naturally occurring
spoken dialogs, Common-ToM, and show that LMs struggle to demonstrate ToM. We
then show that integrating a simple, explicit representation of beliefs
improves LM performance on Common-ToM.

通过自然发生的口语对话构建首个基于一般认知论 (ToM) 的数据集 Common-ToM，研究表明语言模型在表现 ToM 时存在困难，但通过融合简单明确的信念表示可以提高语言模型在 Common-ToM 上的性能。

观点属于我，也属于你：使用共同基础理论对心智进行基准测试

Views Are My Own, But Also Yours: Benchmarking Theory of Mind using  Common Ground

While recent work shows promising results in expanding the capabilities of
large language models (LLM) to directly understand and synthesize speech, an
LLM-based strategy for modeling spoken dialogs remains elusive and calls for
further investigation. This work proposes an extensive speech-text LLM
framework, named the Unified Spoken Dialog Model (USDM), to generate coherent
spoken responses with organic prosodic features relevant to the given input
speech without relying on automatic speech recognition (ASR) or text-to-speech
(TTS) solutions. Our approach employs a multi-step speech-text inference scheme
that leverages chain-of-reasoning capabilities exhibited by the underlying LLM.
We also propose a generalized speech-text pretraining scheme that helps with
capturing cross-modal semantics. Automatic and human evaluations show that the
proposed approach is effective in generating natural-sounding spoken responses,
outperforming both prior and cascaded baselines. Detailed comparative studies
reveal that, despite the cascaded approach being stronger in individual
components, the joint speech-text modeling improves robustness against
recognition errors and speech quality. Demo is available at
this https URL

提出了一个名为统一口语对话模型（USDM）的广泛的语音文本模型框架，用于生成与给定输入语音相关的有机韵律特征的连贯口语回应，而不依赖于自动语音识别（ASR）或文本到语音（TTS）解决方案。该方法利用底层大型语言模型所展示的推理链能力，采用多步骤的语音文本推理方案。经过自动和人工评估表明，该方法在生成自然流畅的口语回应方面非常有效，优于之前的和级联的基线方法。详细的比较研究显示，尽管级联方法在单独的组件上更强大，但联合的语音文本建模改善了对识别错误和语音质量的鲁棒性。