Effectively extracting motions from video is a critical and long-standing
problem for action recognition. This problem is very challenging because
motions (i) do not have an explicit form, (ii) have various concepts such as
displacement, velocity, and acceleration, and (iii) often contain noise caused
by unstable pixels. Addressing these challenges, we propose the Taylor video, a
new video format that highlights the dominate motions (e.g., a waving hand) in
each of its frames named the Taylor frame. Taylor video is named after Taylor
series, which approximates a function at a given point using important terms.
In the scenario of videos, we define an implicit motion-extraction function
which aims to extract motions from video temporal block. In this block, using
the frames, the difference frames, and higher-order difference frames, we
perform Taylor expansion to approximate this function at the starting frame. We
show the summation of the higher-order terms in the Taylor series gives us
dominant motion patterns, where static objects, small and unstable motions are
removed. Experimentally we show that Taylor videos are effective inputs to
popular architectures including 2D CNNs, 3D CNNs, and transformers. When used
individually, Taylor videos yield competitive action recognition accuracy
compared to RGB videos and optical flow. When fused with RGB or optical flow
videos, further accuracy improvement is achieved.

通过使用 Taylor 视频格式和运动提取函数，我们能够高效地从视频中提取动作，并得到具有竞争力的动作识别准确性。

行动识别的泰勒视频

Taylor Videos for Action Recognition

Analyzing human motion is a challenging task with a wide variety of
applications in computer vision and in graphics. One such application, of
particular importance in computer animation, is the retargeting of motion from
one performer to another. While humans move in three dimensions, the vast
majority of human motions are captured using video, requiring 2D-to-3D pose and
camera recovery, before existing retargeting approaches may be applied. In this
paper, we present a new method for retargeting video-captured motion between
different human performers, without the need to explicitly reconstruct 3D poses
and/or camera parameters. In order to achieve our goal, we learn to extract,
directly from a video, a high-level latent motion representation, which is
invariant to the skeleton geometry and the camera view. Our key idea is to
train a deep neural network to decompose temporal sequences of 2D poses into
three components: motion, skeleton, and camera view-angle. Having extracted
such a representation, we are able to re-combine motion with novel skeletons
and camera views, and decode a retargeted temporal sequence, which we compare
to a ground truth from a synthetic dataset. We demonstrate that our framework
can be used to robustly extract human motion from videos, bypassing 3D
reconstruction, and outperforming existing retargeting methods, when applied to
videos in-the-wild. It also enables additional applications, such as
performance cloning, video-driven cartoons, and motion retrieval.

本文提出了一种新的方法，通过使用神经网络从视频中提取人体运动的高级潜在表示的方法，实现了不需要显式重建三维姿态和 / 或相机参数的不同人类表演者之间的视频捕获运动的再定向。

2D 动作重定向中的跨字符动作学习

Learning Character-Agnostic Motion for Motion Retargeting in 2D

We consider the task of learning to extract motion from videos. To this end,
we show that the detection of spatial transformations can be viewed as the
detection of synchrony between the image sequence and a sequence of features
undergoing the motion we wish to detect. We show that learning about synchrony
is possible using very fast, local learning rules, by introducing
multiplicative "gating" interactions between hidden units across frames. This
makes it possible to achieve competitive performance in a wide variety of
motion estimation tasks, using a small fraction of the time required to learn
features, and to outperform hand-crafted spatio-temporal features by a large
margin. We also show how learning about synchrony can be viewed as performing
greedy parameter estimation in the well-known motion energy model.

本篇论文研究了如何学习从视频中提取运动信息。作者提出可以将空间变换检测看作是检测图像序列和运动序列中的 features 同步，提出了一种基于本地乘法门控的快速学习规则，可以在只需花费特征学习时间的一小部分时间内实现在广泛的运动估计任务中竞争性的性能，并且可以通过一定的方法甚至跑过手工处理的时空特征。作者还揭示了学习同步的过程可以被视为在著名的运动能量模型中进行贪婪参数估计。