Post-processing of static embedding has beenshown to improve their
performance on both lexical and sequence-level tasks. However, post-processing
for contextualized embeddings is an under-studied problem. In this work, we
question the usefulness of post-processing for contextualized embeddings
obtained from different layers of pre-trained language models. More
specifically, we standardize individual neuron activations using z-score,
min-max normalization, and by removing top principle components using the
all-but-the-top method. Additionally, we apply unit length normalization to
word representations. On a diverse set of pre-trained models, we show that
post-processing unwraps vital information present in the representations for
both lexical tasks (such as word similarity and analogy)and sequence
classification tasks. Our findings raise interesting points in relation to
theresearch studies that use contextualized representations, and suggest
z-score normalization as an essential step to consider when using them in an
application.

本篇论文探讨了对预训练语言模型中的上下文嵌入层进行后处理的实用性，进一步应用标准化个体神经元激活层、单位长度标准化等方法，发现这些方法能够揭示嵌入层中的重要信息，包括词汇任务和序列分类任务。文章提出了 z-score 标准化作为使用这些方法的一个重要步骤。