While BERT produces high-quality sentence embeddings, its pre-training
computational cost is a significant drawback. In contrast, ELECTRA delivers a
cost-effective pre-training objective and downstream task performance
improvements, but not as performant sentence embeddings. The community tacitly
stopped utilizing ELECTRA's sentence embeddings for semantic textual similarity
(STS). We notice a significant drop in performance when using the ELECTRA
discriminator's last layer in comparison to earlier layers. We explore this
drop and devise a way to repair ELECTRA's embeddings, proposing a novel
truncated model fine-tuning (TMFT) method. TMFT improves the Spearman
correlation coefficient by over 8 points while increasing parameter efficiency
on the STS benchmark dataset. We extend our analysis to various model sizes and
languages. Further, we discover the surprising efficacy of ELECTRA's generator
model, which performs on par with BERT, using significantly fewer parameters
and a substantially smaller embedding size. Finally, we observe further boosts
by combining TMFT with a word similarity task or domain adaptive pre-training.

通过修复 ELECTRA 模型的嵌入方式，提出了一种新的修剪模型微调（TMFT）方法，可以显著提高其在 STS 基准数据集上的 Spearman 相关系数，并增加参数效率；该方法与词相似性任务或领域自适应预训练相结合进一步提升了性能。