Prior work has studied different visual modalities in isolation and developed
separate architectures for recognition of images, videos, and 3D data. Instead,
in this paper, we propose a single model which excels at classifying images,
videos, and single-view 3D data using exactly the same model parameters. Our
'Omnivore' model leverages the flexibility of transformer-based architectures
and is trained jointly on classification tasks from different modalities.
Omnivore is simple to train, uses off-the-shelf standard datasets, and performs
at-par or better than modality-specific models of the same size. A single
Omnivore model obtains 86.0% on ImageNet, 84.1% on Kinetics, and 67.1% on SUN
RGB-D. After finetuning, our models outperform prior work on a variety of
vision tasks and generalize across modalities. Omnivore's shared visual
representation naturally enables cross-modal recognition without access to
correspondences between modalities. We hope our results motivate researchers to
model visual modalities together.

该研究提出了一种基于 Transformer 的 'Omnivore' 模型，使用相同的模型参数在图像、视频、单视角 3D 数据上进行多模态分类，达到与性能同等或更好的效果，并自然地实现了跨模态识别。