How to select relevant key objects and reason about the complex relationships
cross vision and linguistic domain are two key issues in many multi-modality
applications such as visual question answering (VQA). In this work, we
incorporate the visual commonsense information and propose a heterogeneous
graph contrastive learning method to better finish the visual reasoning task.
Our method is designed as a plug-and-play way, so that it can be quickly and
easily combined with a wide range of representative methods. Specifically, our
model contains two key components: the Commonsense-based Contrastive Learning
and the Graph Relation Network. Using contrastive learning, we guide the model
concentrate more on discriminative objects and relevant visual commonsense
attributes. Besides, thanks to the introduction of the Graph Relation Network,
the model reasons about the correlations between homogeneous edges and the
similarities between heterogeneous edges, which makes information transmission
more effective. Extensive experiments on four benchmarks show that our method
greatly improves seven representative VQA models, demonstrating its
effectiveness and generalizability.

本研究通过引入视觉常识信息和提出异构图对比学习方法，改善视觉推理任务，使模型更专注于具有辨别力的对象和相关的视觉常识属性，并通过图关系网络推理同质边之间的关联和异质边之间的相似性，从而提高了模型的信息传输效果。在四个基准测试上的广泛实验表明，本方法显著改进了七个具有代表性的视觉问答模型，证明了它的有效性和泛化性。

基于视觉常识的异构图对比学习

Visual Commonsense based Heterogeneous Graph Contrastive Learning

Understanding emotions during conversation is a fundamental aspect of human
communication, driving NLP research for Emotion Recognition in Conversation
(ERC). While considerable research has focused on discerning emotions of
individual speakers in monolingual dialogues, understanding the emotional
dynamics in code-mixed conversations has received relatively less attention.
This motivates our undertaking of ERC for code-mixed conversations in this
study. Recognizing that emotional intelligence encompasses a comprehension of
worldly knowledge, we propose an innovative approach that integrates
commonsense information with dialogue context to facilitate a deeper
understanding of emotions. To achieve this, we devise an efficient pipeline
that extracts relevant commonsense from existing knowledge graphs based on the
code-mixed input. Subsequently, we develop an advanced fusion technique that
seamlessly combines the acquired commonsense information with the dialogue
representation obtained from a dedicated dialogue understanding module. Our
comprehensive experimentation showcases the substantial performance improvement
obtained through the systematic incorporation of commonsense in ERC. Both
quantitative assessments and qualitative analyses further corroborate the
validity of our hypothesis, reaffirming the pivotal role of commonsense
integration in enhancing ERC.

通过综合运用常识信息与对话理解，我们提出了一种创新方法，能更深入地理解情感，从而通过有效地提取已知知识图表中的相关常识来与对话内容相结合，实现民族语言混合对话中情感识别的实质性性能改进。