Sentence embeddings induced with various transformer architectures encode
much semantic and syntactic information in a distributed manner in a
one-dimensional array. We investigate whether specific grammatical information
can be accessed in these distributed representations. Using data from a task
developed to test rule-like generalizations, our experiments on detecting
subject-verb agreement yield several promising results. First, we show that
while the usual sentence representations encoded as one-dimensional arrays do
not easily support extraction of rule-like regularities, a two-dimensional
reshaping of these vectors allows various learning architectures to access such
information. Next, we show that various architectures can detect patterns in
these two-dimensional reshaped sentence embeddings and successfully learn a
model based on smaller amounts of simpler training data, which performs well on
more complex test data. This indicates that current sentence embeddings contain
information that is regularly distributed, and which can be captured when the
embeddings are reshaped into higher dimensional arrays. Our results cast light
on representations produced by language models and help move towards developing
few-shot learning approaches.

通过使用来自语言模型的更高维数组，我们发现句子嵌入可捕获规则类的语法信息，进而帮助发展少样本学习方法。