Recent studies have shown that sequence-to-sequence (Seq2Seq) models are
limited in solving the compositional generalization (CG) tasks, failing to
systematically generalize to unseen compositions of seen components. There is
mounting evidence that one of the reasons hindering CG is the representation of
the encoder uppermost layer is entangled. In other words, the syntactic and
semantic representations of sequences are twisted inappropriately. However,
most previous studies mainly concentrate on enhancing semantic information at
token-level, rather than composing the syntactic and semantic representations
of sequences appropriately as humans do. In addition, we consider the
representation entanglement problem they found is not comprehensive, and
further hypothesize that source keys and values representations passing into
different decoder layers are also entangled. Staring from this intuition and
inspired by humans' strategies for CG, we propose COMPSITION (Compose Syntactic
and Semantic Representations), an extension to Seq2Seq models to learn to
compose representations of different encoder layers appropriately for
generating different keys and values passing into different decoder layers
through introducing a composed layer between the encoder and decoder.
COMPSITION achieves competitive and even state-of-the-art results on two
realistic benchmarks, which empirically demonstrates the effectiveness of our
proposal.

该研究提出了 COMPSITION 模型，它是 Seq2Seq 模型的一个扩展，旨在通过引入一个复合层来适当地学习不同编码器层的表示，以生成传递到不同解码器层的不同键和值，从而在两个现实基准上取得了有竞争力的甚至是最先进的结果。