Perceiving and understanding non-speech sounds and non-verbal speech is
essential to making decisions that help us interact with our surroundings. In
this paper, we propose GAMA, a novel General-purpose Large Audio-Language Model
(LALM) with Advanced Audio Understanding and Complex Reasoning Abilities. We
build GAMA by integrating an LLM with multiple types of audio representations,
including features from a custom Audio Q-Former, a multi-layer aggregator that
aggregates features from multiple layers of an audio encoder. We fine-tune GAMA
on a large-scale audio-language dataset, which augments it with audio
understanding capabilities. Next, we propose CompA-R (Instruction-Tuning for
Complex Audio Reasoning), a synthetically generated instruction-tuning (IT)
dataset with instructions that require the model to perform complex reasoning
on the input audio. We instruction-tune GAMA with CompA-R to endow it with
complex reasoning abilities, where we further add a soft prompt as input with
high-level semantic evidence by leveraging event tags of the input audio.
Finally, we also propose CompA-R-test, a human-labeled evaluation dataset for
evaluating the capabilities of LALMs on open-ended audio question-answering
that requires complex reasoning. Through automated and expert human
evaluations, we show that GAMA outperforms all other LALMs in literature on
diverse audio understanding tasks by margins of 1%-84%. Further, GAMA IT-ed on
CompA-R proves to be superior in its complex reasoning and instruction
following capabilities.

提出了 GAMA（一种新型通用大型音频语言模型）来解决非语音声音和非语言言语的感知和理解问题。通过集成 LLM 与多种音频表示形式以及利用合成生成的指令调节数据集对其进行训练，实现了音频理解和复杂推理的能力。通过自动化和专家评估，表明 GAMA 在各种音频理解任务中表现优于文献中其他 LALM 模型，具有 1%-84% 的优势。