Vision-Language models (VLMs) have proved effective at aligning image and
text representations, producing superior zero-shot results when transferred to
many downstream tasks. However, these representations suffer some key
shortcomings in Compositional Language Concepts (CLC) understanding such as
recognizing objects' attributes, states, and relations between different
objects. Moreover, VLMs typically have poor interpretability, making it
challenging to debug and mitigate compositional-understanding failures. In this
work, we introduce the Tree-augmented Vision-Language (3VL) model architecture
and training technique accompanied by our proposed Anchor inference method and
Differential Relevance (DiRe) interpretability tool. By expanding the text of
an arbitrary image-text pair into a hierarchical tree structure using language
analysis tools, 3VL allows inducing this structure into the visual
representation learned by the model, enhancing its interpretability and
compositional reasoning. Additionally, we show how Anchor, a simple technique
for text unification, can be employed to filter nuisance factors while
increasing CLC understanding performance, e.g., on the fundamental VL-Checklist
benchmark. We also exhibit how DiRe, which performs a differential comparison
between VLM relevancy maps, enables us to generate compelling visualizations of
the reasons for a model's success or failure.

通过引入树增强视觉语言（3VL）模型架构和训练技术，以及我们提出的锚定推理方法和差分相关性（DiRe）可解释性工具，本研究扩展任意图像 - 文本对的文本内容为分层树状结构，并将该结构引入模型学习的视觉表示，提高了模型的解释能力和组合推理能力，同时展示了锚定和差分相关性工具的应用。