A video autoencoder is proposed for learning disentan- gled representations
of 3D structure and camera pose from videos in a self-supervised manner.
Relying on temporal continuity in videos, our work assumes that the 3D scene
structure in nearby video frames remains static. Given a sequence of video
frames as input, the video autoencoder extracts a disentangled representation
of the scene includ- ing: (i) a temporally-consistent deep voxel feature to
represent the 3D structure and (ii) a 3D trajectory of camera pose for each
frame. These two representations will then be re-entangled for rendering the
input video frames. This video autoencoder can be trained directly using a
pixel reconstruction loss, without any ground truth 3D or camera pose
annotations. The disentangled representation can be applied to a range of
tasks, including novel view synthesis, camera pose estimation, and video
generation by motion following. We evaluate our method on several large- scale
natural video datasets, and show generalization results on out-of-domain
images.

本论文提出了一个自监督学习的视频自编码器，能够从视频中学习出 3D 结构和相机姿态的解耦表示，通过重组这些表示来实现诸如新视角合成、相机姿态估计以及运动跟踪的任务。

视频自编码器：静态三维结构和运动的自我监督解缠

Video Autoencoder: self-supervised disentanglement of static 3D  structure and motion

We describe a new spatio-temporal video autoencoder, based on a classic
spatial image autoencoder and a novel nested temporal autoencoder. The temporal
encoder is represented by a differentiable visual memory composed of
convolutional long short-term memory (LSTM) cells that integrate changes over
time. Here we target motion changes and use as temporal decoder a robust
optical flow prediction module together with an image sampler serving as
built-in feedback loop. The architecture is end-to-end differentiable. At each
time step, the system receives as input a video frame, predicts the optical
flow based on the current observation and the LSTM memory state as a dense
transformation map, and applies it to the current frame to generate the next
frame. By minimising the reconstruction error between the predicted next frame
and the corresponding ground truth next frame, we train the whole system to
extract features useful for motion estimation without any supervision effort.
We present one direct application of the proposed framework in
weakly-supervised semantic segmentation of videos through label propagation
using optical flow.

本文描述一种基于空间图像自编码器和嵌套时间自编码器的新型时空视频自编码器，通过最小化重构误差来训练整个系统，以提取有用于运动估计的特征，其中应用之一为使用光流传播通过标签对视频进行弱监督语义分割。