Natural language programming is a promising approach to enable end users to
instruct new tasks for intelligent agents. However, our formative study found
that end users would often use unclear, ambiguous or vague concepts when
naturally instructing tasks in natural language, especially when specifying
conditionals. Existing systems have limited support for letting the user teach
agents new concepts or explaining unclear concepts. In this paper, we describe
a new multi-modal domain-independent approach that combines natural language
programming and programming-by-demonstration to allow users to first naturally
describe tasks and associated conditions at a high level, and then collaborate
with the agent to recursively resolve any ambiguities or vagueness through
conversations and demonstrations. Users can also define new procedures and
concepts by demonstrating and referring to contents within GUIs of existing
mobile apps. We demonstrate this approach in PUMICE, an end-user programmable
agent that implements this approach. A lab study with 10 users showed its
usability.

本篇论文介绍了一种新的多模态、领域无关的方法，结合自然语言编程和演示编程，允许用户在高层次上自然描述任务和相关条件，并通过对话和演示递归地解决任何模糊不清的问题。PUMICE 是一个可供最终用户编程的代理系统，实现了这种方法。10 位用户的实验显示出系统的可用性。