Contrastive learning has shown promising potential in self-supervised
spatio-temporal representation learning. Most works naively sample different
clips to construct positive and negative pairs. However, we observe that this
formulation inclines the model towards the background scene bias. The
underlying reasons are twofold. First, the scene difference is usually more
noticeable and easier to discriminate than the motion difference. Second, the
clips sampled from the same video often share similar backgrounds but have
distinct motions. Simply regarding them as positive pairs will draw the model
to the static background rather than the motion pattern. To tackle this
challenge, this paper presents a novel dual contrastive formulation.
Concretely, we decouple the input RGB video sequence into two complementary
modes, static scene and dynamic motion. Then, the original RGB features are
pulled closer to the static features and the aligned dynamic features,
respectively. In this way, the static scene and the dynamic motion are
simultaneously encoded into the compact RGB representation. We further conduct
the feature space decoupling via activation maps to distill static- and
dynamic-related features. We term our method as \textbf{D}ual
\textbf{C}ontrastive \textbf{L}earning for spatio-temporal
\textbf{R}epresentation (DCLR). Extensive experiments demonstrate that DCLR
learns effective spatio-temporal representations and obtains state-of-the-art
or comparable performance on UCF-101, HMDB-51, and Diving-48 datasets.

该论文提出了一种新的双重对比学习框架来解决背景场景偏差问题并有效地学习时空表示，实验证明在 UCF-101，HMDB-51 和 Diving-48 数据集上具有最先进的性能。

时空表示的双重对比学习

Dual Contrastive Learning for Spatio-temporal Representation

In self-supervised spatio-temporal representation learning, the temporal
resolution and long-short term characteristics are not yet fully explored,
which limits representation capabilities of learned models. In this paper, we
propose a novel self-supervised method, referred to as video Playback Rate
Perception (PRP), to learn spatio-temporal representation in a
simple-yet-effective way. PRP roots in a dilated sampling strategy, which
produces self-supervision signals about video playback rates for representation
model learning. PRP is implemented with a feature encoder, a classification
module, and a reconstructing decoder, to achieve spatio-temporal semantic
retention in a collaborative discrimination-generation manner. The
discriminative perception model follows a feature encoder to prefer perceiving
low temporal resolution and long-term representation by classifying
fast-forward rates. The generative perception model acts as a feature decoder
to focus on comprehending high temporal resolution and short-term
representation by introducing a motion-attention mechanism. PRP is applied on
typical video target tasks including action recognition and video retrieval.
Experiments show that PRP outperforms state-of-the-art self-supervised models
with significant margins. Code is available at github.com/yuanyao366/PRP

本文提出一种名为 PRP 的自我监督方法，利用扩张采样策略生成关于视频播放速率的监督信号来学习空间时间表示，优于现有的自我监督模型，并应用于动作识别等基本视频任务。

自监督时空表示学习中的视频播放速率感知

Video Playback Rate Perception for Self-supervisedSpatio-Temporal  Representation Learning

For a long time, the vision community tries to learn the spatio-temporal
representation by combining convolutional neural network together with various
temporal models, such as the families of Markov chain, optical flow, RNN and
temporal convolution. However, these pipelines consume enormous computing
resources due to the alternately learning process for spatial and temporal
information. One natural question is whether we can embed the temporal
information into the spatial one so the information in the two domains can be
jointly learned once-only. In this work, we answer this question by presenting
a simple yet powerful operator -- temporal interlacing network (TIN). Instead
of learning the temporal features, TIN fuses the two kinds of information by
interlacing spatial representations from the past to the future, and vice
versa. A differentiable interlacing target can be learned to control the
interlacing process. In this way, a heavy temporal model is replaced by a
simple interlacing operator. We theoretically prove that with a learnable
interlacing target, TIN performs equivalently to the regularized temporal
convolution network (r-TCN), but gains 4% more accuracy with 6x less latency on
6 challenging benchmarks. These results push the state-of-the-art performances
of video understanding by a considerable margin. Not surprising, the ensemble
model of the proposed TIN won the $1^{st}$ place in the ICCV19 - Multi Moments
in Time challenge. Code is made available to facilitate further research at
this https URL

提出了一种称为 TIN（Temporal Interlacing Network）的简单而强大的算子，通过 “交错” 将过去和未来的空间表示融合起来，从而将重型时间模型替换为一个简单的交错算子，以提高视频理解的性能。该算子在 6 个具有挑战性的基准测试中，与正则化时间卷积网络（r-TCN）的性能相当，但在更少的延迟下获得了 4％的增益，极大地推动了视频理解的发展。

时间交错网络

Temporal Interlacing Network

Non-rigid structure-from-motion (NRSfM) has so far been mostly studied for
recovering 3D structure of a single non-rigid/deforming object. To handle the
real world challenging multiple deforming objects scenarios, existing methods
either pre-segment different objects in the scene or treat multiple non-rigid
objects as a whole to obtain the 3D non-rigid reconstruction. However, these
methods fail to exploit the inherent structure in the problem as the solution
of segmentation and the solution of reconstruction could not benefit each
other. In this paper, we propose a unified framework to jointly segment and
reconstruct multiple non-rigid objects. To compactly represent complex
multi-body non-rigid scenes, we propose to exploit the structure of the scenes
along both temporal direction and spatial direction, thus achieving a
spatio-temporal representation. Specifically, we represent the 3D non-rigid
deformations as lying in a union of subspaces along the temporal direction and
represent the 3D trajectories as lying in the union of subspaces along the
spatial direction. This spatio-temporal representation not only provides
competitive 3D reconstruction but also outputs robust segmentation of multiple
non-rigid objects. The resultant optimization problem is solved efficiently
using the Alternating Direction Method of Multipliers (ADMM). Extensive
experimental results on both synthetic and real multi-body NRSfM datasets
demonstrate the superior performance of our proposed framework compared with
the state-of-the-art methods.

本文提出了一种结合分割和重建多个非刚性物体的统一框架，采用时空表示来压缩复杂的多体非刚性场景，可以提供有竞争力的三维重建并输出多个非刚性物体的强鲁棒定位。