In this study, we use the existing Large Language Models ENnhanced to See
Framework (LENS Framework) to test the feasibility of multimodal task-oriented
dialogues. The LENS Framework has been proposed as a method to solve computer
vision tasks without additional training and with fixed parameters of
pre-trained models. We used the Multimodal Dialogs (MMD) dataset, a multimodal
task-oriented dialogue benchmark dataset from the fashion field, and for the
evaluation, we used the ChatGPT-based G-EVAL, which only accepts textual
modalities, with arrangements to handle multimodal data. Compared to
Transformer-based models in previous studies, our method demonstrated an
absolute lift of 10.8% in fluency, 8.8% in usefulness, and 5.2% in relevance
and coherence. The results show that using large-scale models with fixed
parameters rather than using models trained on a dataset from scratch improves
performance in multimodal task-oriented dialogues. At the same time, we show
that Large Language Models (LLMs) are effective for multimodal task-oriented
dialogues. This is expected to lead to efficient applications to existing
systems.

使用现有的 LENS Framework 来测试对话系统的多模态任务的可行性，与以往的基于 Transformer 的模型相比，我们的方法在流畅性、有用性以及相关性和连贯性上均有显著提升，同时证明了大规模预训练模型对于多模态任务导向对话系统的有效性。

冻结的大规模模型在多模态任务导向对话中的应用

Application of frozen large-scale models to multimodal task-oriented  dialogue

Multimodal search-based dialogue is a challenging new task: It extends
visually grounded question answering systems into multi-turn conversations with
access to an external database. We address this new challenge by learning a
neural response generation system from the recently released Multimodal
Dialogue (MMD) dataset (Saha et al., 2017). We introduce a knowledge-grounded
multimodal conversational model where an encoded knowledge base (KB)
representation is appended to the decoder input. Our model substantially
outperforms strong baselines in terms of text-based similarity measures (over 9
BLEU points, 3 of which are solely due to the use of additional information
from the KB.

本研究通过学习 MMD 数据集并实现基于知识库的多模态对话模型解决了多模态基于搜索的对话系统中的新挑战，该模型将编码的知识库表示附加到解码器输入中，从而在文本相似性测量方面获得了高于强基线的表现，其中三个 BLEU 点仅由于使用了来自知识库的附加信息。