We investigate the problem of video Referring Expression Comprehension (REC),
which aims to localize the referent objects described in the sentence to visual
regions in the video frames. Despite the recent progress, existing methods
suffer from two problems: 1) inconsistent localization results across video
frames; 2) confusion between the referent and contextual objects. To this end,
we propose a novel Dual Correspondence Network (dubbed as DCNet) which
explicitly enhances the dense associations in both the inter-frame and
cross-modal manners. Firstly, we aim to build the inter-frame correlations for
all existing instances within the frames. Specifically, we compute the
inter-frame patch-wise cosine similarity to estimate the dense alignment and
then perform the inter-frame contrastive learning to map them close in feature
space. Secondly, we propose to build the fine-grained patch-word alignment to
associate each patch with certain words. Due to the lack of this kind of
detailed annotations, we also predict the patch-word correspondence through the
cosine similarity. Extensive experiments demonstrate that our DCNet achieves
state-of-the-art performance on both video and image REC benchmarks.
Furthermore, we conduct comprehensive ablation studies and thorough analyses to
explore the optimal model designs. Notably, our inter-frame and cross-modal
contrastive losses are plug-and-play functions and are applicable to any video
REC architectures. For example, by building on top of Co-grounding, we boost
the performance by 1.48% absolute improvement on Accu.@0.5 for VID-Sentence
dataset.

研究视频指代表达理解中的两个问题，提出了一种新颖的双重对应网络方法来增强帧间和跨模态的密集关联，以提高视频和图像 REC 基准测试的表现，并进行了全面的剖析研究。

视频指称理解中的对话问题

Correspondence Matters for Video Referring Expression Comprehension

Spatiotemporal predictive learning aims to generate future frames by learning
from historical frames. In this paper, we investigate existing methods and
present a general framework of spatiotemporal predictive learning, in which the
spatial encoder and decoder capture intra-frame features and the middle
temporal module catches inter-frame correlations. While the mainstream methods
employ recurrent units to capture long-term temporal dependencies, they suffer
from low computational efficiency due to their unparallelizable architectures.
To parallelize the temporal module, we propose the Temporal Attention Unit
(TAU), which decomposes the temporal attention into intra-frame statical
attention and inter-frame dynamical attention. Moreover, while the mean squared
error loss focuses on intra-frame errors, we introduce a novel differential
divergence regularization to take inter-frame variations into account.
Extensive experiments demonstrate that the proposed method enables the derived
model to achieve competitive performance on various spatiotemporal prediction
benchmarks.

本文研究了时空预测学习方法，提出了一种通用框架，其中空间编码器和解码器捕获帧内特征，中间的时间模块捕获帧间相关性。我们提出了一种时间注意力单元（TAU）以并行化时间模块，并引入新的差异散度正则化以考虑帧间变化。广泛的实验证明，该方法使得模型在各种时空预测基准测试上具有竞争性能。

时间注意力模块：实现高效时空预测学习

Temporal Attention Unit: Towards Efficient Spatiotemporal Predictive Learning

Skeleton-based Human Activity Recognition has achieved great interest in
recent years as skeleton data has demonstrated being robust to illumination
changes, body scales, dynamic camera views, and complex background. In
particular, Spatial-Temporal Graph Convolutional Networks (ST-GCN) demonstrated
to be effective in learning both spatial and temporal dependencies on
non-Euclidean data such as skeleton graphs. Nevertheless, an effective encoding
of the latent information underlying the 3D skeleton is still an open problem,
especially when it comes to extracting effective information from joint motion
patterns and their correlations. In this work, we propose a novel
Spatial-Temporal Transformer network (ST-TR) which models dependencies between
joints using the Transformer self-attention operator. In our ST-TR model, a
Spatial Self-Attention module (SSA) is used to understand intra-frame
interactions between different body parts, and a Temporal Self-Attention module
(TSA) to model inter-frame correlations. The two are combined in a two-stream
network, whose performance is evaluated on three large-scale datasets,
NTU-RGB+D 60, NTU-RGB+D 120, and Kinetics Skeleton 400, consistently improving
backbone results. Compared with methods that use the same input data, the
proposed ST-TR achieves state-of-the-art performance on all datasets when using
joints' coordinates as input, and results on-par with state-of-the-art when
adding bones information.

本文提出了一种基于 Spatial-Temporal Transformer 网络的骨骼姿态行为识别方法，使用自注意力机制成功提取了骨骼运动及其相关性的信息，与其他同类算法相比在多个大规模数据集上表现出更好的精度。