Next generation task-oriented dialog systems need to understand
conversational contexts with their perceived surroundings, to effectively help
users in the real-world multimodal environment. Existing task-oriented dialog
datasets aimed towards virtual assistance fall short and do not situate the
dialog in the user's multimodal context. To overcome, we present a new dataset
for Situated and Interactive Multimodal Conversations, SIMMC 2.0, which
includes 11K task-oriented user<->assistant dialogs (117K utterances) in the
shopping domain, grounded in immersive and photo-realistic scenes.
The dialogs are collected using a two-phase pipeline: (1) A novel multimodal
dialog simulator generates simulated dialog flows, with an emphasis on
diversity and richness of interactions, (2) Manual paraphrasing of the
generated utterances to collect diverse referring expressions. We provide an
in-depth analysis of the collected dataset, and describe in detail the four
main benchmark tasks we propose. Our baseline model, powered by the
state-of-the-art language model, shows promising results, and highlights new
challenges and directions for the community to study.

本研究提出了一个基于真实情境的多模交互对话数据集 SIMMC 2.0，收集了 11,000 个在购物领域的用户 <-> 助手任务对话。数据集通过模拟器和手工语句改写收集，可用于对话系统的基准测试和自然语言处理研究。