Multi-modal semantic understanding requires integrating information from
different modalities to extract users' real intention behind words. Most
previous work applies a dual-encoder structure to separately encode image and
text, but fails to learn cross-modal feature alignment, making it hard to
achieve cross-modal deep information interaction. This paper proposes a novel
CLIP-guided contrastive-learning-based architecture to perform multi-modal
feature alignment, which projects the features derived from different
modalities into a unified deep space. On multi-modal sarcasm detection (MMSD)
and multi-modal sentiment analysis (MMSA) tasks, the experimental results show
that our proposed model significantly outperforms several baselines, and our
feature alignment strategy brings obvious performance gain over models with
different aggregating methods and models even enriched with knowledge. More
importantly, our model is simple to implement without using task-specific
external knowledge, and thus can easily migrate to other multi-modal tasks. Our
source codes are available at this https URL

本篇研究提出了一种基于 CLIP 引导的对比学习的架构，用于执行多模态特征对齐，将来自不同模态的特征投影到一个统一的深度空间，实验结果表明，我们提出的模型在多模态讽刺检测和多模态情感分析任务中明显优于多个基准模型，我们的特征对齐策略相对于其他聚合方法和甚至富含知识的模型也带来了明显的性能增益，此外，我们的模型实现简单，无需使用特定任务的外部知识，因此可以轻松迁移到其他多模态任务。

多模态语义理解与对比跨模态特征对齐

Multi-modal Semantic Understanding with Contrastive Cross-modal Feature  Alignment

Dual-encoder structure successfully utilizes two language-specific encoders
(LSEs) for code-switching speech recognition. Because LSEs are initialized by
two pre-trained language-specific models (LSMs), the dual-encoder structure can
exploit sufficient monolingual data and capture the individual language
attributes. However, most existing methods have no language constraints on LSEs
and underutilize language-specific knowledge of LSMs. In this paper, we propose
a language-specific characteristic assistance (LSCA) method to mitigate the
above problems. Specifically, during training, we introduce two
language-specific losses as language constraints and generate corresponding
language-specific targets for them. During decoding, we take the decoding
abilities of LSMs into account by combining the output probabilities of two
LSMs and the mixture model to obtain the final predictions. Experiments show
that either the training or decoding method of LSCA can improve the model's
performance. Furthermore, the best result can obtain up to 15.4% relative error
reduction on the code-switching test set by combining the training and decoding
methods of LSCA. Moreover, the system can process code-switching speech
recognition tasks well without extra shared parameters or even retraining based
on two pre-trained LSMs by using our method.

本文提出了一种语言特异性辅助方法（LSCA），用于改善现有方法中 LSEs 的语言约束问题。通过在训练期间引入两个语言特定的损失以及对应的语言特定目标，同时考虑 LSMs 的解码能力，LSCA 方法可以提高模型的性能，最好的结果可以相对误差降低 15.4%。 此外，该系统不需要额外的共享参数或根据两个预训练 LSMs 进行重新训练，可以很好地处理语言切换的语音识别任务。

面向代码切换语音识别的语言特定特征辅助

Language-specific Characteristic Assistance for Code-switching Speech Recognition

Entity retrieval, which aims at disambiguating mentions to canonical entities
from massive KBs, is essential for many tasks in natural language processing.
Recent progress in entity retrieval shows that the dual-encoder structure is a
powerful and efficient framework to nominate candidates if entities are only
identified by descriptions. However, they ignore the property that meanings of
entity mentions diverge in different contexts and are related to various
portions of descriptions, which are treated equally in previous works. In this
work, we propose Multi-View Entity Representations (MuVER), a novel approach
for entity retrieval that constructs multi-view representations for entity
descriptions and approximates the optimal view for mentions via a heuristic
searching method. Our method achieves the state-of-the-art performance on
ZESHEL and improves the quality of candidates on three standard Entity Linking
datasets

该研究提出了一种新方法，称为多视图实体表示（MuVER），旨在提高实体检索的性能，通过启发式搜索方法构造实体描述的多视图表示，并通过这种方式提高候选实体的质量。该方法在 ZESHEL 上取得了最新的性能，并在三个标准实体链接数据集上改善了候选者的质量。