How related are the representations learned by neural language models,
translation models, and language tagging tasks? We answer this question by
adapting an encoder-decoder transfer learning method from computer vision to
investigate the structure among 100 different feature spaces extracted from
hidden representations of various networks trained on language tasks. This
method reveals a low-dimensional structure where language models and
translation models smoothly interpolate between word embeddings, syntactic and
semantic tasks, and future word embeddings. We call this low-dimensional
structure a language representation embedding because it encodes the
relationships between representations needed to process language for a variety
of NLP tasks. We find that this representation embedding can predict how well
each individual feature space maps to human brain responses to natural language
stimuli recorded using fMRI. Additionally, we find that the principal dimension
of this structure can be used to create a metric which highlights the brain's
natural language processing hierarchy. This suggests that the embedding
captures some part of the brain's natural language representation structure.

研究了神经语言模型、翻译模型和语言标注任务中学习到的表示之间的关系，发现了一种低维的语言表示嵌入模型，可以编码处理各种 NLP 任务所需的表示之间的关系，并且可以用来预测各种特征空间与人类大脑对自然语言刺激的响应之间的映射关系，同时主要维度可以用于创建显示大脑的语言处理层次结构的度量。