What is sentence meaning and its ideal representation? Much of the expressive
power of human language derives from semantic composition, the mind's ability
to represent meaning hierarchically & relationally over constituents. At the
same time, much sentential meaning is outside the text and requires grounding
in sensory, motor, and experiential modalities to be adequately learned.
Although large language models display considerable compositional ability,
recent work shows that visually-grounded language models drastically fail to
represent compositional structure. In this thesis, we explore whether & how
models compose visually grounded semantics, and how we might improve their
ability to do so.
Specifically, we introduce 1) WinogroundVQA, a new compositional visual
question answering benchmark, 2) Syntactic Neural Module Distillation, a
measure of compositional ability in sentence embedding models, 3) Causal
Tracing for Image Captioning Models to locate neural representations vital for
vision-language composition, 4) Syntactic MeanPool to inject a compositional
inductive bias into sentence embeddings, and 5) Cross-modal Attention
Congruence Regularization, a self-supervised objective function for
vision-language relation alignment. We close by discussing connections of our
work to neuroscience, psycholinguistics, formal semantics, and philosophy.

本文探讨了视觉图像和语言语义的组合结构问题，并提出了一些方法，如 WinogroundVQA, Syntactic Neural Module Distillation, Causal Tracing for Image Captioning Models,Syntactic MeanPool 和 Cross-modal Attention Congruence Regularization，以提高这种组合结构的能力。