We introduce caption-guided face recognition (CGFR) as a new framework to
improve the performance of commercial-off-the-shelf (COTS) face recognition
(FR) systems. In contrast to combining soft biometrics (eg., facial marks,
gender, and age) with face images, in this work, we use facial descriptions
provided by face examiners as a piece of auxiliary information. However, due to
the heterogeneity of the modalities, improving the performance by directly
fusing the textual and facial features is very challenging, as both lie in
different embedding spaces. In this paper, we propose a contextual feature
aggregation module (CFAM) that addresses this issue by effectively exploiting
the fine-grained word-region interaction and global image-caption association.
Specifically, CFAM adopts a self-attention and a cross-attention scheme for
improving the intra-modality and inter-modality relationship between the image
and textual features, respectively. Additionally, we design a textual feature
refinement module (TFRM) that refines the textual features of the pre-trained
BERT encoder by updating the contextual embeddings. This module enhances the
discriminative power of textual features with a cross-modal projection loss and
realigns the word and caption embeddings with visual features by incorporating
a visual-semantic alignment loss. We implemented the proposed CGFR framework on
two face recognition models (ArcFace and AdaFace) and evaluated its performance
on the Multi-Modal CelebA-HQ dataset. Our framework significantly improves the
performance of ArcFace in both 1:1 verification and 1:N identification
protocol.

引入了一种基于描述指导的人脸识别（CGFR）框架来提高商品化人脸识别系统（COTS FR）性能，通过引入面部描述信息作为辅助信息来改善性能，通过使用上下文特征聚合模块（CFAM）和文本特征细化模块（TFRM）来有效地处理文本和面部特征间的异构性，显著提高了 ArcFace 在多模态 CelebA-HQ 数据集上的验证和识别性能。