We consider the problem of human-machine collaborative problem solving as a
planning task coupled with natural language communication. Our framework
consists of three components -- a natural language engine that parses the
language utterances to a formal representation and vice-versa, a concept
learner that induces generalized concepts for plans based on limited
interactions with the user, and an HTN planner that solves the task based on
human interaction. We illustrate the ability of this framework to address the
key challenges of collaborative problem solving by demonstrating it on a
collaborative building task in a Minecraft-based blocksworld domain. The
accompanied demo video is available at this https URL.

本文介绍了一个人机协同问题解决框架，其中包含自然语言引擎、概念学习器和 HTN 规划器等三个组件，通过在基于 Minecraft 的积木世界中进行合作构建任务来演示该框架解决协同问题解决的关键挑战。

人类辅助的协作问题解决：一种基于自然语言的框架

Human-guided Collaborative Problem Solving: A Natural Language based Framework

In this work, we propose a unified framework, called Visual Reasoning with
Differ-entiable Physics (VRDP), that can jointly learn visual concepts and
infer physics models of objects and their interactions from videos and
language. This is achieved by seamlessly integrating three components: a visual
perception module, a concept learner, and a differentiable physics engine. The
visual perception module parses each video frame into object-centric
trajectories and represents them as latent scene representations. The concept
learner grounds visual concepts (e.g., color, shape, and material) from these
object-centric representations based on the language, thus providing prior
knowledge for the physics engine. The differentiable physics model, implemented
as an impulse-based differentiable rigid-body simulator, performs
differentiable physical simulation based on the grounded concepts to infer
physical properties, such as mass, restitution, and velocity, by fitting the
simulated trajectories into the video observations. Consequently, these learned
concepts and physical models can explain what we have seen and imagine what is
about to happen in future and counterfactual scenarios. Integrating
differentiable physics into the dynamic reasoning framework offers several
appealing benefits. More accurate dynamics prediction in learned physics models
enables state-of-the-art performance on both synthetic and real-world
benchmarks while still maintaining high transparency and interpretability; most
notably, VRDP improves the accuracy of predictive and counterfactual questions
by 4.5% and 11.5% compared to its best counterpart. VRDP is also highly
data-efficient: physical parameters can be optimized from very few videos, and
even a single video can be sufficient. Finally, with all physical parameters
inferred, VRDP can quickly learn new concepts from a few examples.

本文提出了一种名为 VRDP 的统一框架，可以从视频和语言中联合学习视觉概念，并推断对象及其交互的物理模型，通过无缝集成三个组成部分：视觉理解模块、概念学习器和可微分的物理引擎。