We present MUG, a novel interactive task for multimodal grounding where a
user and an agent work collaboratively on an interface screen. Prior works
modeled multimodal UI grounding in one round: the user gives a command and the
agent responds to the command. Yet, in a realistic scenario, a user command can
be ambiguous when the target action is inherently difficult to articulate in
natural language. MUG allows multiple rounds of interactions such that upon
seeing the agent responses, the user can give further commands for the agent to
refine or even correct its actions. Such interaction is critical for improving
grounding performances in real-world use cases. To investigate the problem, we
create a new dataset that consists of 77,820 sequences of human user-agent
interaction on mobile interfaces in which 20% involves multiple rounds of
interactions. To establish our benchmark, we experiment with a range of
modeling variants and evaluation strategies, including both offline and online
evaluation-the online strategy consists of both human evaluation and automatic
with simulators. Our experiments show that allowing iterative interaction
significantly improves the absolute task completion by 18% over the entire test
dataset and 31% over the challenging subset. Our results lay the foundation for
further investigation of the problem.

针对多模态界面对话交互中的语言歧义问题，本文提出了一种新的交互式任务 MUG，并构建了一个包含 77820 组人类用户和智能 Agent 交互的实验数据集，通过离线和在线策略进行评估，实验结果表明允许迭代式交互可以显著提高任务完成率。

MUG: 用户界面上的交互式多模态基础

MUG: Interactive Multimodal Grounding on User Interfaces

In classic instruction following, language like "I'd like the JetBlue flight"
maps to actions (e.g., selecting that flight). However, language also conveys
information about a user's underlying reward function (e.g., a general
preference for JetBlue), which can allow a model to carry out desirable actions
in new contexts. We present a model that infers rewards from language
pragmatically: reasoning about how speakers choose utterances not only to
elicit desired actions, but also to reveal information about their preferences.
On a new interactive flight-booking task with natural language, our model more
accurately infers rewards and predicts optimal actions in unseen environments,
in comparison to past work that first maps language to actions (instruction
following) and then maps actions to rewards (inverse reinforcement learning).

本文提出了一种基于语言认知的模型，不仅能够从语言中推断出用户的奖励函数，以便在新环境中实现期望的操作，而且能够更准确地推断奖励并预测新环境下的最佳操作，相比于以往的指令跟随和逆向强化学习方法。