Multimodal multitask learning has attracted an increasing interest in recent
years. Singlemodal models have been advancing rapidly and have achieved
astonishing results on various tasks across multiple domains. Multimodal
learning offers opportunities for further improvements by integrating data from
multiple modalities. Many methods are proposed to learn on a specific type of
multimodal data, such as vision and language data. A few of them are designed
to handle several modalities and tasks at a time. In this work, we extend and
improve Omninet, an architecture that is capable of handling multiple
modalities and tasks at a time, by introducing cross-cache attention,
integrating patch embeddings for vision inputs, and supporting structured data.
The proposed Structured-data-enhanced Omninet (S-Omninet) is a universal model
that is capable of learning from structured data of various dimensions
effectively with unstructured data through cross-cache attention, which enables
interactions among spatial, temporal, and structured features. We also enhance
spatial representations in a spatial cache with patch embeddings. We evaluate
the proposed model on several multimodal datasets and demonstrate a significant
improvement over the baseline, Omninet.

本文扩展和改进了 Omninet 体系结构，引入了交叉缓存注意力、整合视觉输入的补丁嵌入和支持结构化数据，提出了增强型结构化数据 Omninet (S-Omninet) 模型，能够通过交叉缓存注意力和补丁嵌入在各维度的结构化数据和非结构化数据之间实现有效的相互作用。作者在多模态数据集上评估了所提出的模型，证明相比基准 Omninet 有了显著的性能提升。

S-Omninet: 结构化数据增强的通用多模态学习架构

S-Omninet: Structured Data Enhanced Universal Multimodal Learning  Architecture

Transformer is a popularly used neural network architecture, especially for
language understanding. We introduce an extended and unified architecture that
can be used for tasks involving a variety of modalities like image, text,
videos, etc. We propose a spatio-temporal cache mechanism that enables learning
spatial dimension of the input in addition to the hidden states corresponding
to the temporal input sequence. The proposed architecture further enables a
single model to support tasks with multiple input modalities as well as
asynchronous multi-task learning, thus we refer to it as OmniNet. For example,
a single instance of OmniNet can concurrently learn to perform the tasks of
part-of-speech tagging, image captioning, visual question answering and video
activity recognition. We demonstrate that training these four tasks together
results in about three times compressed model while retaining the performance
in comparison to training them individually. We also show that using this
neural network pre-trained on some modalities assists in learning unseen tasks
such as video captioning and video question answering. This illustrates the
generalization capacity of the self-attention mechanism on the spatio-temporal
cache present in OmniNet.

本文介绍了一种名为 OmniNet 的神经网络框架，使用 Transformer、Spatio-temporal Cache 和自我注意机制来支持多模态学习和异步多任务学习，并演示了训练多个任务的模型在保持性能的前提下大幅度压缩的能力。