Extracting dense representations for terms and phrases is a task of great importance for knowledge discovery platforms targeting highly-technical fields. Dense representations are used as features for downstream components and have multiple applications ranging from ranking results in search to summarization. Common approaches to create dense representations include training domain-specific embeddings with self-supervised setups or using sentence encoder models trained over similarity tasks. In contrast to static embeddings, sentence encoders do not suffer from the out-of-vocabulary (OOV) problem, but impose significant computational costs. In this paper, we propose a fully unsupervised approach to text encoding that consists of training small character-based models with the objective of reconstructing large pre-trained embedding matrices. Models trained with this approach can not only match the quality of sentence encoders in technical domains, but are 5 times smaller and up to 10 times faster, even on high-end GPUs.

本文提出了一种全无监督的文本编码方法，通过训练小型基于字符的模型重构预训练的嵌入矩阵，该方法不仅能够在技术领域匹配句子编码器的质量，而且比后者体积小5倍且速度高达10倍，解决了大规模技术领域下词汇量增加的问题。

从技术领域的术语和短语中提取文本表示