Long-term video generation and prediction remain challenging tasks in
computer vision, particularly in partially observable scenarios where cameras
are mounted on moving platforms. The interaction between observed image frames
and the motion of the recording agent introduces additional complexities. To
address these issues, we introduce the Action-Conditioned Video Generation
(ACVG) framework, a novel approach that investigates the relationship between
actions and generated image frames through a deep dual Generator-Actor
architecture. ACVG generates video sequences conditioned on the actions of
robots, enabling exploration and analysis of how vision and action mutually
influence one another in dynamic environments. We evaluate the framework's
effectiveness on an indoor robot motion dataset which consists of sequences of
image frames along with the sequences of actions taken by the robotic agent,
conducting a comprehensive empirical study comparing ACVG to other
state-of-the-art frameworks along with a detailed ablation study.

介绍了一种新颖的行为条件视频生成框架 (ACVG)，通过深度双发生器 - 行为者结构探究行为与生成的图像帧之间的关系，以机器人的行为为条件生成视频序列，从而在动态环境中探索和分析视觉和行为如何相互影响。通过对室内机器人运动数据集进行全面实证研究和详细消融研究，评估了该框架在长期视频生成中的有效性与其他最先进的框架的比较。

基于行为条件的视频数据提升预测性能

Action-conditioned video data improves predictability

There has been an increasing interest in learning dynamics simulators for
model-based control. Compared with off-the-shelf physics engines, a learnable
simulator can quickly adapt to unseen objects, scenes, and tasks. However,
existing models like interaction networks only work for fully observable
systems; they also only consider pairwise interactions within a single time
step, both restricting their use in practical systems. We introduce Propagation
Networks (PropNet), a differentiable, learnable dynamics model that handles
partially observable scenarios and enables instantaneous propagation of signals
beyond pairwise interactions. Experiments show that our propagation networks
not only outperform current learnable physics engines in forward simulation,
but also achieve superior performance on various control tasks. Compared with
existing model-free deep reinforcement learning algorithms, model-based control
with propagation networks is more accurate, efficient, and generalizable to
new, partially observable scenes and tasks.

本文介绍了 PropNet—— 一种可学习的动力学模型，能够在部分可观测场景下进行模型控制，并实现信号的即时传播。实验结果表明，与其他学习物理引擎相比，PropNet 的前向模拟性能更好，在各种控制任务中也表现出更高的性能。与现有的模型无深度强化学习算法相比，使用 Propagation Networks 实现模型控制更准确、有效，且更具普适性。