Multimodal multitask learning has attracted an increasing interest in recent
years. Singlemodal models have been advancing rapidly and have achieved
astonishing results on various tasks across multiple domains. Multimodal
learning offers opportunities for further improvements by integrating data from
multiple modalities. Many methods are proposed to learn on a specific type of
multimodal data, such as vision and language data. A few of them are designed
to handle several modalities and tasks at a time. In this work, we extend and
improve Omninet, an architecture that is capable of handling multiple
modalities and tasks at a time, by introducing cross-cache attention,
integrating patch embeddings for vision inputs, and supporting structured data.
The proposed Structured-data-enhanced Omninet (S-Omninet) is a universal model
that is capable of learning from structured data of various dimensions
effectively with unstructured data through cross-cache attention, which enables
interactions among spatial, temporal, and structured features. We also enhance
spatial representations in a spatial cache with patch embeddings. We evaluate
the proposed model on several multimodal datasets and demonstrate a significant
improvement over the baseline, Omninet.

本文扩展和改进了 Omninet 体系结构，引入了交叉缓存注意力、整合视觉输入的补丁嵌入和支持结构化数据，提出了增强型结构化数据 Omninet (S-Omninet) 模型，能够通过交叉缓存注意力和补丁嵌入在各维度的结构化数据和非结构化数据之间实现有效的相互作用。作者在多模态数据集上评估了所提出的模型，证明相比基准 Omninet 有了显著的性能提升。