Past work probing compositionality in sentence embedding models faces issues
determining the causal impact of implicit syntax representations. Given a
sentence, we construct a neural module net based on its syntax parse and train
it end-to-end to approximate the sentence's embedding generated by a
transformer model. The distillability of a transformer to a Syntactic NeurAl
Module Net (SynNaMoN) then captures whether syntax is a strong causal model of
its compositional ability. Furthermore, we address questions about the geometry
of semantic composition by specifying individual SynNaMoN modules' internal
architecture & linearity. We find differences in the distillability of various
sentence embedding models that broadly correlate with their performance, but
observe that distillability doesn't considerably vary by model size. We also
present preliminary evidence that much syntax-guided composition in sentence
embedding models is linear, and that non-linearities may serve primarily to
handle non-compositional phrases.

通过构建基于语法解析的神经模块网络，研究表明语法是句子嵌入模型合成能力中的强因果模型，同时发现句子嵌入模型的可深化性与性能相关，而非线性的语法引导合成只在处理非组成性短语时起作用。