Learning new skills by observing humans' behaviors is an essential capability
of AI. In this work, we leverage instructional videos to study humans'
decision-making processes, focusing on learning a model to plan goal-directed
actions in real-life videos. In contrast to conventional action recognition,
goal-directed actions are based on expectations of their outcomes requiring
causal knowledge of potential consequences of actions. Thus, integrating the
environment structure with goals is critical for solving this task. Previous
works learn a single world model will fail to distinguish various tasks,
resulting in an ambiguous latent space; planning through it will gradually
neglect the desired outcomes since the global information of the future goal
degrades quickly as the procedure evolves. We address these limitations with a
new formulation of procedure planning and propose novel algorithms to model
human behaviors through Bayesian Inference and model-based Imitation Learning.
Experiments conducted on real-world instructional videos show that our method
can achieve state-of-the-art performance in reaching the indicated goals.
Furthermore, the learned contextual information presents interesting features
for planning in a latent space.

本文提出了一种新的过程规划公式，通过贝叶斯推论和基于模型的模仿学习，建模人类行为，从而在实际的指导视频中实现了它，证明了我们的方法可以实现达到指定目标的最先进性能的同时，学习到的上下文信息以潜在空间的形式表现出有趣的特征。

通过情境建模和基于模型的策略学习在教学视频中进行程序规划

Procedure Planning in Instructional Videos via Contextual Modeling and  Model-based Policy Learning

The ability to plan and execute goal specific actions in varied, unexpected
settings is a central requirement of intelligent agents. In this paper, we
explore how an agent can be equipped with an internal model of the dynamics of
the external world, and how it can use this model to plan novel actions by
running multiple internal simulations ("visual imagination"). Our models
directly process raw visual input, and use a novel object-centric prediction
formulation based on visual glimpses centered on objects (fixations) to enforce
translational invariance of the learned physical laws. The agent gathers
training data through random interaction with a collection of different
environments, and the resulting model can then be used to plan goal-directed
actions in novel environments that the agent has not seen before. We
demonstrate that our agent can accurately plan actions for playing a simulated
billiards game, which requires pushing a ball into a target position or into
collision with another ball.

本文中，我们探讨了一种方法，能够让智能体装备内部的世界动态模型，并通过运行多个内部仿真（“视觉想象”）来规划新型动作。我们的模型直接处理原始视觉输入，并使用基于目标中心的视觉快照（聚焦）的新颖物体中心预测公式来强制学习物理定律的平移不变性。我们的智能体可以通过随机与不同环境的集合进行交互来收集训练数据，然后可以使用所得到的模型在未见过的环境中规划目标导向的行动。我们展示了我们的智能体可以准确地规划模拟台球游戏中的行动，该游戏需要将球推向目标位置或使其与其他球碰撞。