This study aims at improving the performance of scoring student responses in
science education automatically. BERT-based language models have shown
significant superiority over traditional NLP models in various language-related
tasks. However, science writing of students, including argumentation and
explanation, is domain-specific. In addition, the language used by students is
different from the language in journals and Wikipedia, which are training
sources of BERT and its existing variants. All these suggest that a
domain-specific model pre-trained using science education data may improve
model performance. However, the ideal type of data to contextualize pre-trained
language model and improve the performance in automatically scoring student
written responses remains unclear. Therefore, we employ different data in this
study to contextualize both BERT and SciBERT models and compare their
performance on automatic scoring of assessment tasks for scientific
argumentation. We use three datasets to pre-train the model: 1) journal
articles in science education, 2) a large dataset of students' written
responses (sample size over 50,000), and 3) a small dataset of students'
written responses of scientific argumentation tasks. Our experimental results
show that in-domain training corpora constructed from science questions and
responses improve language model performance on a wide variety of downstream
tasks. Our study confirms the effectiveness of continual pre-training on
domain-specific data in the education domain and demonstrates a generalizable
strategy for automating science education tasks with high accuracy. We plan to
release our data and SciEdBERT models for public use and community engagement.

本研究旨在提高科学教育中学生答案自动评分的性能。使用科学教育数据预训练特定领域的语言模型可以提高性能，并在将来自动化科学教育任务方面具有潜在的普适性。

上下文至关重要：一种针对科学教育预训练语言模型的策略

Context Matters: A Strategy to Pre-train Language Model for Science Education

Open Domain Question Answering requires systems to retrieve external
knowledge and perform multi-hop reasoning by composing knowledge spread over
multiple sentences. In the recently introduced open domain question answering
challenge datasets, QASC and OpenBookQA, we need to perform retrieval of facts
and compose facts to correctly answer questions. In our work, we learn a
semantic knowledge ranking model to re-rank knowledge retrieved through Lucene
based information retrieval systems. We further propose a "knowledge fusion
model" which leverages knowledge in BERT-based language models with externally
retrieved knowledge and improves the knowledge understanding of the BERT-based
language models. On both OpenBookQA and QASC datasets, the knowledge fusion
model with semantically re-ranked knowledge outperforms previous attempts.

本文提出一种基于语义知识排名模型和知识融合模型的开放领域问答方法，该方法通过整合基于 Lucene 的信息检索系统检索的知识和 BERT 语言模型中的外部检索知识，以正确回答问题。研究表明，基于知识融合模型的方法优于之前的尝试。