With the growing demand for developing speech-based interaction models, end-to-end Spoken Language Models (SLMs) have emerged as a promising solution. When engaging in conversations with humans, it is essential for these models to comprehend a wide range of world knowledge. In this paper, we introduce VoxEval, a novel speech question-answering benchmark specifically designed to assess SLMs' knowledge understanding through purely speech-based interactions. Unlike existing AudioQA benchmarks, VoxEval maintains speech format for both questions and answers, evaluates model robustness across diverse audio conditions (varying timbres, audio qualities, and speaking styles), and pioneers the assessment of challenging domains like mathematical problem-solving in spoken format. Our comprehensive evaluation of recent SLMs using VoxEval reveals significant performance limitations in current models, highlighting crucial areas for future improvements.

本文针对当前端到端语音语言模型在知识理解方面的不足，提出了VoxEval，一个新的基于语音的问答基准，专注于通过语音交互评估模型的知识理解能力。研究发现，现有模型在多样化音频条件下表现出显著的性能限制，为未来改进提供了关键方向。

VoxEval：评估端到端语音语言模型的知识理解能力