Current RGB-D scene recognition approaches often train two standalone
backbones for RGB and depth modalities with the same Places or ImageNet
pre-training. However, the pre-trained depth network is still biased by
RGB-based models which may result in a suboptimal solution. In this paper, we
present a single-model self-supervised hybrid pre-training framework for RGB
and depth modalities, termed as CoMAE. Our CoMAE presents a curriculum learning
strategy to unify the two popular self-supervised representation learning
algorithms: contrastive learning and masked image modeling. Specifically, we
first build a patch-level alignment task to pre-train a single encoder shared
by two modalities via cross-modal contrastive learning. Then, the pre-trained
contrastive encoder is passed to a multi-modal masked autoencoder to capture
the finer context features from a generative perspective. In addition, our
single-model design without requirement of fusion module is very flexible and
robust to generalize to unimodal scenario in both training and testing phases.
Extensive experiments on SUN RGB-D and NYUDv2 datasets demonstrate the
effectiveness of our CoMAE for RGB and depth representation learning. In
addition, our experiment results reveal that CoMAE is a data-efficient
representation learner. Although we only use the small-scale and unlabeled
training set for pre-training, our CoMAE pre-trained models are still
competitive to the state-of-the-art methods with extra large-scale and
supervised RGB dataset pre-training. Code will be released at
this https URL

本文提出了一种名为 CoMAE 的单模型自监督混合预训练框架，通过交叉模态对比学习和遮蔽图像建模，采用课程学习策略来统一两种流行的自监督表示学习算法，并设计了一个修补程序级别对齐任务来预训练一种单一的编码器，共享两个模态。CoMAE 在 SUN RGB-D 和 NYUDv2 数据集上的实验表明，在仅使用小规模和无标签的训练集进行预训练的情况下，其预训练模型仍然具有与额外的大规模和监督 RGB 数据集预训练的最新方法相竞争的效果。

CoMAE：基于单模型的小规模 RGB-D 数据混合预训练

CoMAE: Single Model Hybrid Pre-training on Small-Scale RGB-D Datasets

The capacity of empathy is crucial to the success of open-domain dialog
systems. Due to its nature of multi-dimensionality, there are various factors
that relate to empathy expression, such as communication mechanism, dialog act
and emotion. However, existing methods for empathetic response generation
usually either consider only one empathy factor or ignore the hierarchical
relationships between different factors, leading to a weak ability of empathy
modeling. In this paper, we propose a multi-factor hierarchical framework,
CoMAE, for empathetic response generation, which models the above three key
factors of empathy expression in a hierarchical way. We show experimentally
that our CoMAE-based model can generate more empathetic responses than previous
methods. We also highlight the importance of hierarchical modeling of different
factors through both the empirical analysis on a real-life corpus and the
extensive experiments. Our codes and used data are available at
this https URL

本文提出了一种多因素分层框架 CoMAE，用于模拟情感应答生成功能，实验表明，我们的模型比之前的方法表现更好，分层建模不同因素的重要性也得到了验证。