Large language models (LLMs) trained on general domain corpora showed
remarkable results on natural language processing (NLP) tasks. However,
previous research demonstrated LLMs trained using domain-focused corpora
perform better on specialized tasks. Inspired by this pivotal insight, we
developed INDUS, a comprehensive suite of LLMs tailored for the Earth science,
biology, physics, heliophysics, planetary sciences and astrophysics domains and
trained using curated scientific corpora drawn from diverse data sources. The
suite of models include: (1) an encoder model trained using domain-specific
vocabulary and corpora to address natural language understanding tasks, (2) a
contrastive-learning-based general text embedding model trained using a diverse
set of datasets drawn from multiple sources to address information retrieval
tasks and (3) smaller versions of these models created using knowledge
distillation techniques to address applications which have latency or resource
constraints. We also created three new scientific benchmark datasets namely,
CLIMATE-CHANGE-NER (entity-recognition), NASA-QA (extractive QA) and NASA-IR
(IR) to accelerate research in these multi-disciplinary fields. Finally, we
show that our models outperform both general-purpose encoders (RoBERTa) and
existing domain-specific encoders (SciBERT) on these new tasks as well as
existing benchmark tasks in the domains of interest.

我们开发了 INDUS，一套针对地球科学、生物学、物理学、太阳物理学、行星科学和天体物理学领域进行定制的大型语言模型（LLMs），并使用从多种数据源中提取的策划科学文献进行训练。此模型套件包括利用特定领域词汇和文献进行训练的编码器模型，基于对比学习的一般文本嵌入模型，以及使用知识蒸馏技术创建的资源受限应用模型。我们还创建了三个新的科学基准数据集，加速研究这些多学科领域，最后证明我们的模型在这些新任务以及感兴趣领域的现有基准任务上优于通用编码器（RoBERTa）和现有的领域特定编码器（SciBERT）。