Large language models (LLMs) have exhibited remarkable capabilities across a
variety of domains and tasks, challenging our understanding of learning and
cognition. Despite the recent success, current LLMs are not capable of
processing complex audio information or conducting spoken conversations (like
Siri or Alexa). In this work, we propose a multi-modal AI system named
AudioGPT, which complements LLMs (i.e., ChatGPT) with 1) foundation models to
process complex audio information and solve numerous understanding and
generation tasks; and 2) the input/output interface (ASR, TTS) to support
spoken dialogue. With an increasing demand to evaluate multi-modal LLMs of
human intention understanding and cooperation with foundation models, we
outline the principles and processes and test AudioGPT in terms of consistency,
capability, and robustness. Experimental results demonstrate the capabilities
of AudioGPT in solving AI tasks with speech, music, sound, and talking head
understanding and generation in multi-round dialogues, which empower humans to
create rich and diverse audio content with unprecedented ease. Our system is
publicly available at https://github.com/AIGC-Audio/AudioGPT.

本文提出了一种名为 AudioGPT 的多模 AI 系统，该系统结合了基础模型来处理复杂的音频信息和解决许多理解和生成任务，以及支持口语对话的输入 / 输出接口（ASR，TTS），并通过一系列实验证明了 AudioGPT 在多轮对话中具有语音、音乐、声音和对话理解和生成任务的能力。