We investigate what kind of structural knowledge learned in neural network
encoders is transferable to processing natural language. We design artificial
languages with structural properties that mimic natural language, pretrain
encoders on the data, and see how much performance the encoder exhibits on
downstream tasks in natural language. Our experimental results show that
pretraining with an artificial language with a nesting dependency structure
provides some knowledge transferable to natural language. A follow-up probing
analysis indicates that its success in the transfer is related to the amount of
encoded contextual information and what is transferred is the knowledge of
position-aware context dependence of language. Our results provide insights
into how neural network encoders process human languages and the source of
cross-lingual transferability of recent multilingual language models.

通过在人工语言上预训练编码器并探究其在自然语言处理中的表现，研究神经网络编码器中可迁移的结构知识，发现嵌套依赖结构的人工语言的预训练可以提供一些可迁移的知识，并且转移的成功与编码器中编码的上下文信息量相关，转移的是语言上下文依赖的位置感知知识。