In this paper, we explore the capability of an agent to construct a logical
sequence of action steps, thereby assembling a strategic procedural plan. This
plan is crucial for navigating from an initial visual observation to a target
visual outcome, as depicted in real-life instructional videos. Existing works
have attained partial success by extensively leveraging various sources of
information available in the datasets, such as heavy intermediate visual
observations, procedural names, or natural language step-by-step instructions,
for features or supervision signals. However, the task remains formidable due
to the implicit causal constraints in the sequencing of steps and the
variability inherent in multiple feasible plans. To tackle these intricacies
that previous efforts have overlooked, we propose to enhance the capabilities
of the agent by infusing it with procedural knowledge. This knowledge, sourced
from training procedure plans and structured as a directed weighted graph,
equips the agent to better navigate the complexities of step sequencing and its
potential variations. We coin our approach KEPP, a novel Knowledge-Enhanced
Procedure Planning system, which harnesses a probabilistic procedural knowledge
graph extracted from training data, effectively acting as a comprehensive
textbook for the training domain. Experimental evaluations across three
widely-used datasets under settings of varying complexity reveal that KEPP
attains superior, state-of-the-art results while requiring only minimal
supervision.

该研究探讨了代理程序在构建逻辑动作步骤序列方面的能力，从而组装战略性的过程计划。利用来自训练数据的程序化知识图，我们提出了一种名为 KEPP 的新型知识增强程序规划系统，该系统在各种复杂度设置下的实验评估结果表明，KEPP 取得了卓越的、最新的结果，同时只需要很少的监督。

为何不使用教科书？知识增强的教学视频程序规划

Why Not Use Your Textbook? Knowledge-Enhanced Procedure Planning of  Instructional Videos

In this paper, we consider domain-adaptive imitation learning with visual
observation, where an agent in a target domain learns to perform a task by
observing expert demonstrations in a source domain. Domain adaptive imitation
learning arises in practical scenarios where a robot, receiving visual sensory
data, needs to mimic movements by visually observing other robots from
different angles or observing robots of different shapes. To overcome the
domain shift in cross-domain imitation learning with visual observation, we
propose a novel framework for extracting domain-independent behavioral features
from input observations that can be used to train the learner, based on dual
feature extraction and image reconstruction. Empirical results demonstrate that
our approach outperforms previous algorithms for imitation learning from visual
observation with domain shift.

在这篇论文中，我们考虑了具有视觉观测的领域自适应模仿学习，其中目标领域中的一个代理通过观察源领域中的专家示范来学习执行任务。我们提出了一种新的框架，通过双重特征提取和图像重构从输入观测中提取领域无关的行为特征，以克服视觉观察中跨领域模仿学习中的领域转移问题。实证结果表明，我们的方法优于先前的具有领域转移的视觉观察模仿学习算法。

领域自适应视觉观察的模仿学习

Domain Adaptive Imitation Learning with Visual Observation

Intelligent agents can learn to represent the action spaces of other agents
simply by observing them act. Such representations help agents quickly learn to
predict the effects of their own actions on the environment and to plan complex
action sequences. In this work, we address the problem of learning an agent's
action space purely from visual observation. We use stochastic video prediction
to learn a latent variable that captures the scene's dynamics while being
minimally sensitive to the scene's static content. We introduce a loss term
that encourages the network to capture the composability of visual sequences
and show that it leads to representations that disentangle the structure of
actions. We call the full model with composable action representations
Composable Learned Action Space Predictor (CLASP). We show the applicability of
our method to synthetic settings and its potential to capture action spaces in
complex, realistic visual settings. When used in a semi-supervised setting, our
learned representations perform comparably to existing fully supervised methods
on tasks such as action-conditioned video prediction and planning in the
learned action space, while requiring orders of magnitude fewer action labels.
Project website: this https URL

本研究使用随机视频预测学习了捕捉场景动态的潜在变量，同时尽量不受场景内容的影响，从而学习代理的行动空间；该方法在半监督学习下表现相当于现有的完全监督方法，在任务如行动条件视频预测和计划学习行动空间时，需要几个数量级更少的行动标签。