Stochastic video prediction enables the consideration of uncertainty in
future motion, thereby providing a better reflection of the dynamic nature of
the environment. Stochastic video prediction methods based on image
auto-regressive recurrent models need to feed their predictions back into the
latent space. Conversely, the state-space models, which decouple frame
synthesis and temporal prediction, proves to be more efficient. However,
inferring long-term temporal information about motion and generalizing to
dynamic scenarios under non-stationary assumptions remains an unresolved
challenge. In this paper, we propose a state-space decomposition stochastic
video prediction model that decomposes the overall video frame generation into
deterministic appearance prediction and stochastic motion prediction. Through
adaptive decomposition, the model's generalization capability to dynamic
scenarios is enhanced. In the context of motion prediction, obtaining a prior
on the long-term trend of future motion is crucial. Thus, in the stochastic
motion prediction branch, we infer the long-term motion trend from conditional
frames to guide the generation of future frames that exhibit high consistency
with the conditional frames. Experimental results demonstrate that our model
outperforms baselines on multiple datasets.

通过自适应分解的方法，我们提出了一种状态空间分解的随机视频预测模型，将整体视频帧生成分解为确定性外观预测和随机运动预测，从而增强模型对动态场景的泛化能力。实验结果表明，我们的模型在多个数据集上优于基线模型。

考虑长期运动趋势的视频预测的状态空间分解模型

State-space Decomposition Model for Video Prediction Considering  Long-term Motion Trend

While stochastic video prediction models enable future prediction under
uncertainty, they mostly fail to model the complex dynamics of real-world
scenes. For example, they cannot provide reliable predictions for scenes with a
moving camera and independently moving foreground objects in driving scenarios.
The existing methods fail to fully capture the dynamics of the structured world
by only focusing on changes in pixels. In this paper, we assume that there is
an underlying process creating observations in a video and propose to factorize
it into static and dynamic components. We model the static part based on the
scene structure and the ego-motion of the vehicle, and the dynamic part based
on the remaining motion of the dynamic objects. By learning separate
distributions of changes in foreground and background, we can decompose the
scene into static and dynamic parts and separately model the change in each.
Our experiments demonstrate that disentangling structure and motion helps
stochastic video prediction, leading to better future predictions in complex
driving scenarios on two real-world driving datasets, KITTI and Cityscapes.

本文提出了将视频观察结果分解为静态和动态部分，分别建模于场景结构和车辆主体运动、动态物体运动，来分离结构和运动的影响，从而更好地预测复杂驾驶场景中的未来运动。

结构与运动的随机视频预测

Stochastic Video Prediction with Structure and Motion

Designing video prediction models that account for the inherent uncertainty
of the future is challenging. Most works in the literature are based on
stochastic image-autoregressive recurrent networks, which raises several
performance and applicability issues. An alternative is to use fully latent
temporal models which untie frame synthesis and temporal dynamics. However, no
such model for stochastic video prediction has been proposed in the literature
yet, due to design and training difficulties. In this paper, we overcome these
difficulties by introducing a novel stochastic temporal model whose dynamics
are governed in a latent space by a residual update rule. This first-order
scheme is motivated by discretization schemes of differential equations. It
naturally models video dynamics as it allows our simpler, more interpretable,
latent model to outperform prior state-of-the-art methods on challenging
datasets.

介绍了一种基于残差更新规则的新型随机时间模型，可用于视频预测，并在复杂数据集上胜过先前的最先进方法。