We present Any-Modality Augmented Language Model (AnyMAL), a unified model
that reasons over diverse input modality signals (i.e. text, image, video,
audio, IMU motion sensor), and generates textual responses. AnyMAL inherits the
powerful text-based reasoning abilities of the state-of-the-art LLMs including
LLaMA-2 (70B), and converts modality-specific signals to the joint textual
space through a pre-trained aligner module. To further strengthen the
multimodal LLM's capabilities, we fine-tune the model with a multimodal
instruction set manually collected to cover diverse topics and tasks beyond
simple QAs. We conduct comprehensive empirical analysis comprising both human
and automatic evaluations, and demonstrate state-of-the-art performance on
various multimodal tasks.

我们提出了 Any-Modality Augmented Language Model (AnyMAL)，这是一个统一模型，可以对多样化的输入模态信号（文本、图像、视频、音频、IMU 运动传感器）进行推理，并生成文本回复。AnyMAL 继承了最先进的 LLMs（如 LLaMA-2 (70B)）的强大的基于文本的推理能力，并通过预训练的对齐模块将模态特定信号转换为联合文本空间。为了进一步增强多模态 LLM 的功能，我们使用人工收集的多模态指令集对模型进行了微调，以覆盖复杂的主题和任务。我们进行了全面的经验分析，包括人工和自动评估，并在各种多模态任务上展示了最先进的性能。