Effectively extracting motions from video is a critical and long-standing
problem for action recognition. This problem is very challenging because
motions (i) do not have an explicit form, (ii) have various concepts such as
displacement, velocity, and acceleration, and (iii) often contain noise caused
by unstable pixels. Addressing these challenges, we propose the Taylor video, a
new video format that highlights the dominate motions (e.g., a waving hand) in
each of its frames named the Taylor frame. Taylor video is named after Taylor
series, which approximates a function at a given point using important terms.
In the scenario of videos, we define an implicit motion-extraction function
which aims to extract motions from video temporal block. In this block, using
the frames, the difference frames, and higher-order difference frames, we
perform Taylor expansion to approximate this function at the starting frame. We
show the summation of the higher-order terms in the Taylor series gives us
dominant motion patterns, where static objects, small and unstable motions are
removed. Experimentally we show that Taylor videos are effective inputs to
popular architectures including 2D CNNs, 3D CNNs, and transformers. When used
individually, Taylor videos yield competitive action recognition accuracy
compared to RGB videos and optical flow. When fused with RGB or optical flow
videos, further accuracy improvement is achieved.

通过使用 Taylor 视频格式和运动提取函数，我们能够高效地从视频中提取动作，并得到具有竞争力的动作识别准确性。