Current multimodal and multitask foundation models like 4M or UnifiedIO show
promising results, but in practice their out-of-the-box abilities to accept
diverse inputs and perform diverse tasks are limited by the (usually rather
small) number of modalities and tasks they are trained on. In this paper, we
expand upon the capabilities of them by training a single model on tens of
highly diverse modalities and by performing co-training on large-scale
multimodal datasets and text corpora. This includes training on several
semantic and geometric modalities, feature maps from recent state of the art
models like DINOv2 and ImageBind, pseudo labels of specialist models like SAM
and 4DHumans, and a range of new modalities that allow for novel ways to
interact with the model and steer the generation, for example image metadata or
color palettes. A crucial step in this process is performing discrete
tokenization on various modalities, whether they are image-like, neural network
feature maps, vectors, structured data like instance segmentation or human
poses, or data that can be represented as text. Through this, we expand on the
out-of-the-box capabilities of multimodal models and specifically show the
possibility of training one model to solve at least 3x more tasks/modalities
than existing ones and doing so without a loss in performance. This enables
more fine-grained and controllable multimodal generation capabilities and
allows us to study the distillation of models trained on diverse data and
objectives into a unified model. We successfully scale the training to a three
billion parameter model using tens of modalities and different datasets. The
resulting models and training code are open sourced at 4m.epfl.ch.

通过对多个多媒体形式进行联合训练和扩展，我们成功地提升了多模态模型的能力，并展示了训练一个模型以解决更多任务 / 多模态性的可能性，而不损失性能，从而实现更精细化和可控的多模态生成能力。