Majority of research in learning based methods has been towards designing and
training networks for specific tasks. However, many of the learning based
tasks, across modalities, share commonalities and could be potentially tackled
in a joint framework. We present an approach in such direction, to learn
multiple tasks, in multiple modalities, with a unified architecture. The
proposed network is composed of task specific encoders, a common trunk in the
middle, followed by task specific prediction heads. We first pre-train it by
self-supervised masked training, followed by sequential training for the
different tasks. We train the network on all major modalities, e.g.\ visual,
audio, text and 3D, and report results on $22$ diverse and challenging public
benchmarks. We demonstrate empirically that, using a joint network to train
across modalities leads to meaningful information sharing and this allows us to
achieve state-of-the-art results on most of the benchmarks. We also show
generalization of the trained network on cross-modal tasks as well as unseen
datasets and tasks.

学习基于任务的方法共享模态的共同之处，共同架构多任务多模态的网络，使得跨模态任务的联合训练能够实现信息共享并获得最先进的结果。

OmniVec：跨模态共享学习的鲁棒表示学习

OmniVec: Learning robust representations with cross modal sharing

Human parsing and pose estimation have recently received considerable
interest due to their substantial application potentials. However, the existing
datasets have limited numbers of images and annotations and lack a variety of
human appearances and coverage of challenging cases in unconstrained
environments. In this paper, we introduce a new benchmark named "Look into
Person (LIP)" that provides a significant advancement in terms of scalability,
diversity, and difficulty, which are crucial for future developments in
human-centric analysis. This comprehensive dataset contains over 50,000
elaborately annotated images with 19 semantic part labels and 16 body joints,
which are captured from a broad range of viewpoints, occlusions, and background
complexities. Using these rich annotations, we perform detailed analyses of the
leading human parsing and pose estimation approaches, thereby obtaining
insights into the successes and failures of these methods. To further explore
and take advantage of the semantic correlation of these two tasks, we propose a
novel joint human parsing and pose estimation network to explore efficient
context modeling, which can simultaneously predict parsing and pose with
extremely high quality. Furthermore, we simplify the network to solve human
parsing by exploring a novel self-supervised structure-sensitive learning
approach, which imposes human pose structures into the parsing results without
resorting to extra supervision. The dataset, code and models are available at
this http URL

本文提出了一个名为 “Look into Person (LIP)” 的新数据集，其包含超过 50,000 张精细注释的图像，利用这些注释，作者进行了人体解析、姿势估计方法的详细分析，并提出了一种新的联合人体解析和姿势估计网络以及简化网络以解决人体解析的方法。