We present a systematic investigation of layer-wise BERT activations for
general-purpose text representations to understand what linguistic information
they capture and how transferable they are across different tasks.
Sentence-level embeddings are evaluated against two state-of-the-art models on
downstream and probing tasks from SentEval, while passage-level embeddings are
evaluated on four question-answering (QA) datasets under a learning-to-rank
problem setting. Embeddings from the pre-trained BERT model perform poorly in
semantic similarity and sentence surface information probing tasks. Fine-tuning
BERT on natural language inference data greatly improves the quality of the
embeddings. Combining embeddings from different BERT layers can further boost
performance. BERT embeddings outperform BM25 baseline significantly on factoid
QA datasets at the passage level, but fail to perform better than BM25 on
non-factoid datasets. For all QA datasets, there is a gap between
embedding-based method and in-domain fine-tuned BERT (we report new
state-of-the-art results on two datasets), which suggests deep interactions
between question and answer pairs are critical for those hard tasks.

该论文系统地研究了面向通用文本表示的分层 BERT 激活，以了解其捕获的语言信息以及它们在不同任务之间的可转移性。在下游和探测任务中，句子级别的嵌入与两个最先进的模型进行了比较，而段落级别的嵌入则在四个问答（QA）数据集上进行了学习排名问题设置的评估。结果表明，将预训练的 BERT 模型微调于自然语言推断数据上可以显著提高嵌入的质量。