One of the current state-of-the-art multilingual document embedding model
LASER is based on the bidirectional LSTM neural machine translation model. This
paper presents a transformer-based sentence/document embedding model, T-LASER,
which makes three significant improvements. Firstly, the BiLSTM layers is
replaced by the attention-based transformer layers, which is more capable of
learning sequential patterns in longer texts. Secondly, due to the absence of
recurrence, T-LASER enables faster parallel computations in the encoder to
generate the text embedding. Thirdly, we augment the NMT translation loss
function with an additional novel distance constraint loss. This distance
constraint loss would further bring the embeddings of parallel sentences close
together in the vector space; we call the T-LASER model trained with distance
constraint, cT-LASER. Our cT-LASER model significantly outperforms both
BiLSTM-based LASER and the simpler transformer-based T-LASER.

本研究基于双向 LSTM 神经机器翻译模型探索了基于 transformer 的句子 / 文档嵌入模型 T-LASER，并使用距离约束损失函数增强 NMT 损失函数，提升模型在多语言文本处理方面的性能。