Learning effective sentence representations is crucial for many Natural
Language Processing (NLP) tasks, including semantic search, semantic textual
similarity (STS), and clustering. While multiple transformer models have been
developed for sentence embedding learning, these models may not perform
optimally when dealing with specialized domains like aviation, which has unique
characteristics such as technical jargon, abbreviations, and unconventional
grammar. Furthermore, the absence of labeled datasets makes it difficult to
train models specifically for the aviation domain. To address these challenges,
we propose a novel approach for adapting sentence transformers for the aviation
domain. Our method is a two-stage process consisting of pre-training followed
by fine-tuning. During pre-training, we use Transformers and Sequential
Denoising AutoEncoder (TSDAE) with aviation text data as input to improve the
initial model performance. Subsequently, we fine-tune our models using a
Natural Language Inference (NLI) dataset in the Sentence Bidirectional Encoder
Representations from Transformers (SBERT) architecture to mitigate overfitting
issues. Experimental results on several downstream tasks show that our adapted
sentence transformers significantly outperform general-purpose transformers,
demonstrating the effectiveness of our approach in capturing the nuances of the
aviation domain. Overall, our work highlights the importance of domain-specific
adaptation in developing high-quality NLP solutions for specialized industries
like aviation.

针对航空领域的技术术语和非传统语法等特征，作者提出了一种两阶段的方法：使用航空文本数据进行预训练，然后使用自然语言推理数据集在 SBERT 架构上进行微调，以适应该领域并显著优于通用性转换器，突出了领域特定适应对于开发专业 NLP 解决方案的重要性。