Generative Pre-trained Transformer (GPT) models have achieved remarkable
performance on various natural language processing tasks. However, there has
been limited research on applying similar frameworks to audio tasks. Previously
proposed large language models for audio tasks either lack sufficient
quantitative evaluations, or are limited to tasks for recognizing and
understanding audio content, or significantly underperform existing
state-of-the-art (SOTA) models. In this paper, we propose LauraGPT, a unified
GPT model for audio recognition, understanding, and generation. LauraGPT is a
versatile language model that can process both audio and text inputs and
generate outputs in either modalities. It can perform a wide range of tasks
related to content, semantics, paralinguistics, and audio-signal analysis. Some
of its noteworthy tasks include automatic speech recognition, speech-to-text
translation, text-to-speech synthesis, machine translation, speech enhancement,
automated audio captioning, speech emotion recognition, and spoken language
understanding. To achieve this goal, we use a combination of continuous and
discrete features for audio. We encode input audio into continuous
representations using an audio encoder and decode output audio from discrete
codec codes. We then fine-tune a large decoder-only Transformer-based language
model on multiple audio-to-text, text-to-audio, audio-to-audio, and
text-to-text tasks using a supervised multitask learning approach. Extensive
experiments show that LauraGPT achieves competitive or superior performance
compared to existing SOTA models on various audio processing benchmarks.

本文提出了 LauraGPT，一个统一的 GPT 模型，用于音频识别、理解和生成，实现了在多个音频处理基准测试上与现有 SOTA 模型相媲美或卓越的性能。