Audio-visual understanding is a rapidly evolving field that seeks to integrate and interpret information from both auditory and visual modalities. Despite recent advances in multi-modal learning, existing benchmarks often suffer from strong visual bias -- where answers can be inferred from visual data alone -- and provide only aggregate scores that conflate multiple sources of error. This makes it difficult to determine whether models struggle with visual understanding, audio interpretation, or audio-visual alignment. In this work, we introduce DAVE (Diagnostic Audio Visual Evaluation), a novel benchmark dataset designed to systematically evaluate audio-visual models across controlled challenges. DAVE alleviates existing limitations by (i) ensuring both modalities are necessary to answer correctly and (ii) decoupling evaluation into atomic subcategories. Our detailed analysis of state-of-the-art models reveals specific failure modes and provides targeted insights for improvement. By offering this standardized diagnostic framework, we aim to facilitate more robust development of audio-visual models. The dataset is released: https://github.com/gorjanradevski/dave

本研究针对音视频理解领域中现有基准测试的视觉偏见问题，提出了一种新颖的基准数据集DAVE（诊断音视频评估），旨在系统性地评估音视频模型。DAVE通过确保两种模态在回答时都必不可少来克服现有局限性，并将评估解耦为原子子类别，揭示了最新模型的特定失败模式，为改进提供了有针对性的见解。

诊断基准：音视频评估的DAVE