We present MatSci-NLP, a natural language benchmark for evaluating the performance of natural language processing (NLP) models on materials science text. We construct the benchmark from publicly available materials science text data to encompass seven different NLP tasks, including conventional NLP tasks like named entity recognition and relation classification, as well as NLP tasks specific to materials science, such as synthesis action retrieval which relates to creating synthesis procedures for materials. We study various BERT-based models pretrained on different scientific text corpora on MatSci-NLP to understand the impact of pretraining strategies on understanding materials science text. Given the scarcity of high-quality annotated data in the materials science domain, we perform our fine-tuning experiments with limited training data to encourage the generalize across MatSci-NLP tasks. Our experiments in this low-resource training setting show that language models pretrained on scientific text outperform BERT trained on general text. MatBERT, a model pretrained specifically on materials science journals, generally performs best for most tasks. Moreover, we propose a unified text-to-schema for multitask learning on \benchmark and compare its performance with traditional fine-tuning methods. In our analysis of different training methods, we find that our proposed text-to-schema methods inspired by question-answering consistently outperform single and multitask NLP fine-tuning methods. The code and datasets are publicly available at \url{https://github.com/BangLab-UdeM-Mila/NLP4MatSci-ACL23}.

本文提出了MatSci-NLP标准，旨在评估自然语言处理模型在材料科学文本上的性能，其中包括命名实体识别和关系分类等传统NLP任务以及涉及创建材料合成程序的特定于材料科学的NLP任务。作者的实验结果表明，预先训练的模型对科学文本的理解优于对一般文本的BERT模型训练，并提出了一种基于问题回答方法的NLP多任务学习方法，其性能优于传统的微调方法。

MatSci-NLP: 使用文本到模式建模评估科学语言模型在材料科学语言任务上的表现