Multimodal intent recognition poses significant challenges, requiring the
incorporation of non-verbal modalities from real-world contexts to enhance the
comprehension of human intentions. Existing benchmark datasets are limited in
scale and suffer from difficulties in handling out-of-scope samples that arise
in multi-turn conversational interactions. We introduce MIntRec2.0, a
large-scale benchmark dataset for multimodal intent recognition in multi-party
conversations. It contains 1,245 dialogues with 15,040 samples, each annotated
within a new intent taxonomy of 30 fine-grained classes. Besides 9,304 in-scope
samples, it also includes 5,736 out-of-scope samples appearing in multi-turn
contexts, which naturally occur in real-world scenarios. Furthermore, we
provide comprehensive information on the speakers in each utterance, enriching
its utility for multi-party conversational research. We establish a general
framework supporting the organization of single-turn and multi-turn dialogue
data, modality feature extraction, multimodal fusion, as well as in-scope
classification and out-of-scope detection. Evaluation benchmarks are built
using classic multimodal fusion methods, ChatGPT, and human evaluators. While
existing methods incorporating nonverbal information yield improvements,
effectively leveraging context information and detecting out-of-scope samples
remains a substantial challenge. Notably, large language models exhibit a
significant performance gap compared to humans, highlighting the limitations of
machine learning methods in the cognitive intent understanding task. We believe
that MIntRec2.0 will serve as a valuable resource, providing a pioneering
foundation for research in human-machine conversational interactions, and
significantly facilitating related applications. The full dataset and codes are
available at this https URL

MIntRec2.0 是一个大规模的多模态意图识别基准数据集，包含 1,245 个对话，15,040 个样本，使用 30 个细粒度类别的新意图分类法进行注释，还包括 5,736 个出现在多轮对话上下文中的超范围样本。

MIntRec2.0：一个用于对话中多模态意图识别和超范围检测的大规模基准数据集

MIntRec2.0: A Large-scale Benchmark Dataset for Multimodal Intent  Recognition and Out-of-scope Detection in Conversations

Multimodal intent recognition aims to leverage diverse modalities such as
expressions, body movements and tone of speech to comprehend user's intent,
constituting a critical task for understanding human language and behavior in
real-world multimodal scenarios. Nevertheless, the majority of existing methods
ignore potential correlations among different modalities and own limitations in
effectively learning semantic features from nonverbal modalities. In this
paper, we introduce a token-level contrastive learning method with
modality-aware prompting (TCL-MAP) to address the above challenges. To
establish an optimal multimodal semantic environment for text modality, we
develop a modality-aware prompting module (MAP), which effectively aligns and
fuses features from text, video and audio modalities with similarity-based
modality alignment and cross-modality attention mechanism. Based on the
modality-aware prompt and ground truth labels, the proposed token-level
contrastive learning framework (TCL) constructs augmented samples and employs
NT-Xent loss on the label token. Specifically, TCL capitalizes on the optimal
textual semantic insights derived from intent labels to guide the learning
processes of other modalities in return. Extensive experiments show that our
method achieves remarkable improvements compared to state-of-the-art methods.
Additionally, ablation analyses demonstrate the superiority of the
modality-aware prompt over the handcrafted prompt, which holds substantial
significance for multimodal prompt learning. The codes are released at
this https URL

引入了一个名为 TCL-MAP 的方法，通过引导其它模态学习过程，利用文本语义洞察力来提高多模态意图识别的效果。该方法在实验证明了其在准确度上相较于现有方法有显著提升，而模态感知提示对多模态提示学习具有重要意义。

多模态意图识别的令牌级对比学习与模态感知提示

Token-Level Contrastive Learning with Modality-Aware Prompting for  Multimodal Intent Recognition

Multimodal intent recognition is a significant task for understanding human
language in real-world multimodal scenes. Most existing intent recognition
methods have limitations in leveraging the multimodal information due to the
restrictions of the benchmark datasets with only text information. This paper
introduces a novel dataset for multimodal intent recognition (MIntRec) to
address this issue. It formulates coarse-grained and fine-grained intent
taxonomies based on the data collected from the TV series Superstore. The
dataset consists of 2,224 high-quality samples with text, video, and audio
modalities and has multimodal annotations among twenty intent categories.
Furthermore, we provide annotated bounding boxes of speakers in each video
segment and achieve an automatic process for speaker annotation. MIntRec is
helpful for researchers to mine relationships between different modalities to
enhance the capability of intent recognition. We extract features from each
modality and model cross-modal interactions by adapting three powerful
multimodal fusion methods to build baselines. Extensive experiments show that
employing the non-verbal modalities achieves substantial improvements compared
with the text-only modality, demonstrating the effectiveness of using
multimodal information for intent recognition. The gap between the
best-performing methods and humans indicates the challenge and importance of
this task for the community. The full dataset and codes are available for use
at this https URL.

这篇论文介绍了一个新的数据集（MIntRec）来解决现有意图识别方法在实现多模态信息利用方面的限制。使用该数据集，通过提取每种模态的特征并采用三种强大的多模态融合方法，可以显著提高意图识别的效果。