Next generation task-oriented dialog systems need to understand
conversational contexts with their perceived surroundings, to effectively help
users in the real-world multimodal environment. Existing task-oriented dialog
datasets aimed towards virtual assistance fall short and do not situate the
dialog in the user's multimodal context. To overcome, we present a new dataset
for Situated and Interactive Multimodal Conversations, SIMMC 2.0, which
includes 11K task-oriented user<->assistant dialogs (117K utterances) in the
shopping domain, grounded in immersive and photo-realistic scenes.
The dialogs are collected using a two-phase pipeline: (1) A novel multimodal
dialog simulator generates simulated dialog flows, with an emphasis on
diversity and richness of interactions, (2) Manual paraphrasing of the
generated utterances to collect diverse referring expressions. We provide an
in-depth analysis of the collected dataset, and describe in detail the four
main benchmark tasks we propose. Our baseline model, powered by the
state-of-the-art language model, shows promising results, and highlights new
challenges and directions for the community to study.

本研究提出了一个基于真实情境的多模交互对话数据集 SIMMC 2.0，收集了 11,000 个在购物领域的用户 <-> 助手任务对话。数据集通过模拟器和手工语句改写收集，可用于对话系统的基准测试和自然语言处理研究。

SIMMC 2.0：面向任务的沉浸式多模态对话数据集

SIMMC 2.0: A Task-oriented Dialog Dataset for Immersive Multimodal  Conversations

The growing popularity of Virtual Assistants poses new challenges for Entity
Resolution, the task of linking mentions in text to their referent entities in
a knowledge base. Specifically, in the shopping domain, customers tend to use
implicit utterances (e.g., "organic milk") rather than explicit names, leading
to a large number of candidate products. Meanwhile, for the same query,
different customers may expect different results. For example, with "add milk
to my cart", a customer may refer to a certain organic product, while some
customers may want to re-order products they regularly purchase. To address
these issues, we propose a new framework that leverages personalized features
to improve the accuracy of product ranking. We first build a cross-source
heterogeneous knowledge graph from customer purchase history and product
knowledge graph to jointly learn customer and product embeddings. After that,
we incorporate product, customer, and history representations into a neural
reranking model to predict which candidate is most likely to be purchased for a
specific customer. Experiments show that our model substantially improves the
accuracy of the top ranked candidates by 24.6% compared to the state-of-the-art
product search model.

提出了一个利用个性化特征提高产品排名准确性的新框架，该框架通过建立跨来源异构知识图来联合学习客户和产品嵌入，并将产品、客户和历史表示结合到神经重排模型中以预测特定客户最有可能购买的候选产品。实验表明，相比最先进的产品搜索模型，我们的模型将前几名的候选产品的准确性提高了 24.6%。