We present MUG, a novel interactive task for multimodal grounding where a
user and an agent work collaboratively on an interface screen. Prior works
modeled multimodal UI grounding in one round: the user gives a command and the
agent responds to the command. Yet, in a realistic scenario, a user command can
be ambiguous when the target action is inherently difficult to articulate in
natural language. MUG allows multiple rounds of interactions such that upon
seeing the agent responses, the user can give further commands for the agent to
refine or even correct its actions. Such interaction is critical for improving
grounding performances in real-world use cases. To investigate the problem, we
create a new dataset that consists of 77,820 sequences of human user-agent
interaction on mobile interfaces in which 20% involves multiple rounds of
interactions. To establish our benchmark, we experiment with a range of
modeling variants and evaluation strategies, including both offline and online
evaluation-the online strategy consists of both human evaluation and automatic
with simulators. Our experiments show that allowing iterative interaction
significantly improves the absolute task completion by 18% over the entire test
dataset and 31% over the challenging subset. Our results lay the foundation for
further investigation of the problem.

针对多模态界面对话交互中的语言歧义问题，本文提出了一种新的交互式任务 MUG，并构建了一个包含 77820 组人类用户和智能 Agent 交互的实验数据集，通过离线和在线策略进行评估，实验结果表明允许迭代式交互可以显著提高任务完成率。