We propose MAMo, a novel memory and attention frame-work for monocular video
depth estimation. MAMo can augment and improve any single-image depth
estimation networks into video depth estimation models, enabling them to take
advantage of the temporal information to predict more accurate depth. In MAMo,
we augment model with memory which aids the depth prediction as the model
streams through the video. Specifically, the memory stores learned visual and
displacement tokens of the previous time instances. This allows the depth
network to cross-reference relevant features from the past when predicting
depth on the current frame. We introduce a novel scheme to continuously update
the memory, optimizing it to keep tokens that correspond with both the past and
the present visual information. We adopt attention-based approach to process
memory features where we first learn the spatio-temporal relation among the
resultant visual and displacement memory tokens using self-attention module.
Further, the output features of self-attention are aggregated with the current
visual features through cross-attention. The cross-attended features are
finally given to a decoder to predict depth on the current frame. Through
extensive experiments on several benchmarks, including KITTI, NYU-Depth V2, and
DDAD, we show that MAMo consistently improves monocular depth estimation
networks and sets new state-of-the-art (SOTA) accuracy. Notably, our MAMo video
depth estimation provides higher accuracy with lower latency, when omparing to
SOTA cost-volume-based video depth models.

我们提出了 MAMo，这是一种新颖的内存和注意力框架，用于单目视频深度估计。MAMo 可以将任何单幅图像深度估计网络增强和改进为视频深度估计模型，使其利用时间信息以预测更准确的深度。通过在视频中流式传输的模型中增加内存，MAMo 在深度预测中起到辅助作用。具体而言，内存储存了先前时间实例的学习到的视觉和位移标记。这使得深度网络能够在预测当前帧深度时参考过去的相关特征。我们引入了一种新的方案来持续更新内存，通过优化保留与过去和当前视觉信息相对应的标记。我们采用基于注意力的方法来处理内存特征，其中我们首先使用自注意模块学习得到了生成的视觉和位移内存标记之间的时空关系。接下来，通过交叉注意力，将自注意力的输出特征与当前的视觉特征相结合。最后，将交叉注意力的特征输入到解码器中，以预测当前帧的深度。通过对 KITTI、NYU-Depth V2 和 DDAD 等多个基准测试的大量实验证明，MAMo 始终改进了单目深度估计网络，并建立了新的最先进准确性标准。值得注意的是，与基于代价体积的最先进视频深度模型相比，我们的 MAMo 视频深度估计在更低延迟的情况下提供了更高的准确性。

MAMo：利用记忆和注意力进行单目视频深度估计

MAMo: Leveraging Memory and Attention for Monocular Video Depth  Estimation

Multimodal representation learning has shown promising improvements on
various vision-language tasks. Most existing methods excel at building
global-level alignment between vision and language while lacking effective
fine-grained image-text interaction. In this paper, we propose a jointly masked
multimodal modeling method to learn fine-grained multimodal representations.
Our method performs joint masking on image-text input and integrates both
implicit and explicit targets for the masked signals to recover. The implicit
target provides a unified and debiased objective for vision and language, where
the model predicts latent multimodal representations of the unmasked input. The
explicit target further enriches the multimodal representations by recovering
high-level and semantically meaningful information: momentum visual features of
image patches and concepts of word tokens. Through such a masked modeling
process, our model not only learns fine-grained multimodal interaction, but
also avoids the semantic gap between high-level representations and low- or
mid-level prediction targets (e.g. image pixels), thus producing semantically
rich multimodal representations that perform well on both zero-shot and
fine-tuned settings. Our pre-trained model (named MAMO) achieves
state-of-the-art performance on various downstream vision-language tasks,
including image-text retrieval, visual question answering, visual reasoning,
and weakly-supervised visual grounding.

本文提出一种联合掩蔽多模态建模方法 (MAMO)，通过联合掩盖图像 - 文本输入，并通过隐式和显式目标来恢复掩蔽信号，从而学习细粒度的多模态表示，实现高级和语义明确的信息恢复，取得了各种下游视觉 - 语言任务中的最新成果。