In this work, we explore a scalable way for building a general representation
model toward unlimited modalities. We release ONE-PEACE, a highly extensible
model with 4B parameters that can seamlessly align and integrate
representations across vision, audio, and language modalities. The architecture
of ONE-PEACE comprises modality adapters, shared self-attention layers, and
modality FFNs. This design allows for the easy extension of new modalities by
adding adapters and FFNs, while also enabling multi-modal fusion through
self-attention layers. To pretrain ONE-PEACE, we develop two modality-agnostic
pretraining tasks, cross-modal aligning contrast and intra-modal denoising
contrast, which align the semantic space of different modalities and capture
fine-grained details within modalities concurrently. With the scaling-friendly
architecture and pretraining tasks, ONE-PEACE has the potential to expand to
unlimited modalities. Without using any vision or language pretrained model for
initialization, ONE-PEACE achieves leading results on a wide range of uni-modal
and multi-modal tasks, including image classification (ImageNet), semantic
segmentation (ADE20K), audio-text retrieval (AudioCaps, Clotho), audio
classification (ESC-50, FSD50K, VGGSound), audio question answering (AVQA),
image-text retrieval (MSCOCO, Flickr30K), and visual grounding (RefCOCO/+/g).
Code is available at this https URL

本文提出了一种可扩展的模型来对多种感知模式进行整合，其中 ONE-PEACE 是一个高度可扩展的模型，通过跨感谢模态对其表示进行无缝整合，并通过多模态融合实现对其进行扩展，无需使用任何视觉或语言预训练模型初始化，并在各种单模态和多模态任务上取得了领先的结果。