Cutting-edge research in facial expression recognition (FER) currently favors
the utilization of convolutional neural networks (CNNs) backbone which is
supervisedly pre-trained on face recognition datasets for feature extraction.
However, due to the vast scale of face recognition datasets and the high cost
associated with collecting facial labels, this pre-training paradigm incurs
significant expenses. Towards this end, we propose to pre-train vision
Transformers (ViTs) through a self-supervised approach on a mid-scale general
image dataset. In addition, when compared with the domain disparity existing
between face datasets and FER datasets, the divergence between general datasets
and FER datasets is more pronounced. Therefore, we propose a contrastive
fine-tuning approach to effectively mitigate this domain disparity.
Specifically, we introduce a novel FER training paradigm named Mask Image
pre-training with MIx Contrastive fine-tuning (MIMIC). In the initial phase, we
pre-train the ViT via masked image reconstruction on general images.
Subsequently, in the fine-tuning stage, we introduce a mix-supervised
contrastive learning process, which enhances the model with a more extensive
range of positive samples by the mixing strategy. Through extensive experiments
conducted on three benchmark datasets, we demonstrate that our MIMIC
outperforms the previous training paradigm, showing its capability to learn
better representations. Remarkably, the results indicate that the vanilla ViT
can achieve impressive performance without the need for intricate,
auxiliary-designed modules. Moreover, when scaling up the model size, MIMIC
exhibits no performance saturation and is superior to the current
state-of-the-art methods.

利用自监督学习方法在中等规模的图像数据集上，对视觉 Transformer 进行预训练，并通过对比微调的方式有效减少面部数据集和面部表情识别数据集之间的领域差异，实现更好的表征学习。

MIMIC：蒙面图像预训练并混合对比细调用于面部表情识别

MIMIC: Mask Image Pre-training with Mix Contrastive Fine-tuning for  Facial Expression Recognition

The performance of sentence encoders can be significantly improved through
the simple practice of fine-tuning using contrastive loss. A natural question
arises: what characteristics do models acquire during contrastive learning?
This paper theoretically and experimentally shows that contrastive-based
sentence encoders implicitly weight words based on information-theoretic
quantities; that is, more informative words receive greater weight, while
others receive less. The theory states that, in the lower bound of the optimal
value of the contrastive learning objective, the norm of word embedding
reflects the information gain associated with the distribution of surrounding
words. We also conduct comprehensive experiments using various models, multiple
datasets, two methods to measure the implicit weighting of models (Integrated
Gradients and SHAP), and two information-theoretic quantities (information gain
and self-information). The results provide empirical evidence that contrastive
fine-tuning emphasizes informative words.

通过对比损失的简单调优，句子编码器的性能可以得到显著提高。本文理论上和实验上展示出，在对比学习过程中模型获得了怎样的特征，即基于信息论的数量对词汇进行隐式加权。本文还使用多种模型、多个数据集、两种度量模型隐式加权的方法（集成渐变法和 SHAP）以及两个信息论数量（信息增益和自信息量）进行了全面实验。结果提供了对对比调优强调信息丰富词汇的经验证据。

基于对比学习的句子编码器隐式加权信息词汇

Contrastive Learning-based Sentence Encoders Implicitly Weight  Informative Words

Fine-grained classification involves dealing with datasets with larger number
of classes with subtle differences between them. Guiding the model to focus on
differentiating dimensions between these commonly confusable classes is key to
improving performance on fine-grained tasks. In this work, we analyse the
contrastive fine-tuning of pre-trained language models on two fine-grained text
classification tasks, emotion classification and sentiment analysis. We
adaptively embed class relationships into a contrastive objective function to
help differently weigh the positives and negatives, and in particular,
weighting closely confusable negatives more than less similar negative
examples. We find that Label-aware Contrastive Loss outperforms previous
contrastive methods, in the presence of larger number and/or more confusable
classes, and helps models to produce output distributions that are more
differentiated.

本文分析了在细粒度文本分类任务情感分类和情感分析中预训练语言模型的对比微调。适应性嵌入类关系的对比目标函数可以有助于不同地权衡正面和负面情感，使得 Label-aware Contrastive Loss 在更多混淆类别的情况下表现更好。