In this study, we use the existing Large Language Models ENnhanced to See
Framework (LENS Framework) to test the feasibility of multimodal task-oriented
dialogues. The LENS Framework has been proposed as a method to solve computer
vision tasks without additional training and with fixed parameters of
pre-trained models. We used the Multimodal Dialogs (MMD) dataset, a multimodal
task-oriented dialogue benchmark dataset from the fashion field, and for the
evaluation, we used the ChatGPT-based G-EVAL, which only accepts textual
modalities, with arrangements to handle multimodal data. Compared to
Transformer-based models in previous studies, our method demonstrated an
absolute lift of 10.8% in fluency, 8.8% in usefulness, and 5.2% in relevance
and coherence. The results show that using large-scale models with fixed
parameters rather than using models trained on a dataset from scratch improves
performance in multimodal task-oriented dialogues. At the same time, we show
that Large Language Models (LLMs) are effective for multimodal task-oriented
dialogues. This is expected to lead to efficient applications to existing
systems.

使用现有的 LENS Framework 来测试对话系统的多模态任务的可行性，与以往的基于 Transformer 的模型相比，我们的方法在流畅性、有用性以及相关性和连贯性上均有显著提升，同时证明了大规模预训练模型对于多模态任务导向对话系统的有效性。

冻结的大规模模型在多模态任务导向对话中的应用

Application of frozen large-scale models to multimodal task-oriented  dialogue

SimpleMTOD is a simple language model which recasts several sub-tasks in
multimodal task-oriented dialogues as sequence prediction tasks. SimpleMTOD is
built on a large-scale transformer-based auto-regressive architecture, which
has already proven to be successful in uni-modal task-oriented dialogues, and
effectively leverages transfer learning from pre-trained GPT-2. In-order to
capture the semantics of visual scenes, we introduce both local and
de-localized tokens for objects within a scene. De-localized tokens represent
the type of an object rather than the specific object itself and so possess a
consistent meaning across the dataset. SimpleMTOD achieves a state-of-the-art
BLEU score (0.327) in the Response Generation sub-task of the SIMMC 2.0
test-std dataset while performing on par in other multimodal sub-tasks:
Disambiguation, Coreference Resolution, and Dialog State Tracking. This is
despite taking a minimalist approach for extracting visual (and non-visual)
information. In addition the model does not rely on task-specific architectural
changes such as classification heads.

SimpleMTOD 提出了一个简单的语言模型，将多模态任务导向对话的几个子任务转换为序列预测任务，在大规模的基于 transformer 的自回归体系结构上构建，引入了局部和去局部化令牌来捕获视觉场景的语义，并在不依赖于分类头等任务特定体系结构的情况下，在 SIMMC 2.0 测试数据集 Response Generation 子任务中实现了最先进的 BLEU 分数（0.327）。