Visual Entailment with natural language explanations aims to infer the
relationship between a text-image pair and generate a sentence to explain the
decision-making process. Previous methods rely mainly on a pre-trained
vision-language model to perform the relation inference and a language model to
generate the corresponding explanation. However, the pre-trained
vision-language models mainly build token-level alignment between text and
image yet ignore the high-level semantic alignment between the phrases (chunks)
and visual contents, which is critical for vision-language reasoning. Moreover,
the explanation generator based only on the encoded joint representation does
not explicitly consider the critical decision-making points of relation
inference. Thus the generated explanations are less faithful to visual-language
reasoning. To mitigate these problems, we propose a unified Chunk-aware
Alignment and Lexical Constraint based method, dubbed as CALeC. It contains a
Chunk-aware Semantic Interactor (arr. CSI), a relation inferrer, and a Lexical
Constraint-aware Generator (arr. LeCG). Specifically, CSI exploits the sentence
structure inherent in language and various image regions to build chunk-aware
semantic alignment. Relation inferrer uses an attention-based reasoning network
to incorporate the token-level and chunk-level vision-language representations.
LeCG utilizes lexical constraints to expressly incorporate the words or chunks
focused by the relation inferrer into explanation generation, improving the
faithfulness and informativeness of the explanations. We conduct extensive
experiments on three datasets, and experimental results indicate that CALeC
significantly outperforms other competitor models on inference accuracy and
quality of generated explanations.

本文提出了一种名为 CALeC 的方法，旨在解决语言和图像理解中，以前的方法在语义和各种特征之间的对齐上存在问题，其包括一个基于块意识和词汇约束的生成器和一个基于关系的推理网络，用于提高视觉语言理解的精确度和生成的解释的质量。在三个数据集的广泛实验中，CALeC 明显优于其他竞争模型，可以准确推断出 text-image 对之间的关系并生成解释。