We extract mathematical concepts from mathematical text using generative
large language models (LLMs) like ChatGPT, contributing to the field of
automatic term extraction (ATE) and mathematical text processing, and also to
the study of LLMs themselves. Our work builds on that of others in that we aim
for automatic extraction of terms (keywords) in one mathematical field,
category theory, using as a corpus the 755 abstracts from a snapshot of the
online journal "Theory and Applications of Categories", circa 2020. Where our
study diverges from previous work is in (1) providing a more thorough analysis
of what makes mathematical term extraction a difficult problem to begin with;
(2) paying close attention to inter-annotator disagreements; (3) providing a
set of guidelines which both human and machine annotators could use to
standardize the extraction process; (4) introducing a new annotation tool to
help humans with ATE, applicable to any mathematical field and even beyond
mathematics; (5) using prompts to ChatGPT as part of the extraction process,
and proposing best practices for such prompts; and (6) raising the question of
whether ChatGPT could be used as an annotator on the same level as human
experts. Our overall findings are that the matter of mathematical ATE is an
interesting field which can benefit from participation by LLMs, but LLMs
themselves cannot at this time surpass human performance on it.

我们使用 ChatGPT 等生成性大型语言模型从数学文本中提取数学概念，为自动术语提取和数学文本处理领域以及 LLM 研究作出贡献。我们的研究目标是在数学领域中通过自动提取术语（关键词）来建立模型，使用的语料库是在线期刊《Theory and Applications of Categories》的 755 个摘要，快照来自 2020 年左右。我们的研究与之前的工作不同之处在于：（1）更彻底分析了数学术语提取的困难之处；（2）关注了人工标注者之间的不一致性；（3）提供了一套人机标注者都可以使用的指南，以规范化提取过程；（4）引入了一种新的注释工具，帮助人类进行自动术语提取，适用于任何数学领域甚至数学之外的领域；（5）使用 ChatGPT 的提示作为提取过程的一部分，并提出了最佳实践；（6）提出了 ChatGPT 是否能够作为与人类专家在同一级别上的注释者的问题。我们的总体发现是，数学术语提取是一个有趣的领域，LLM 的参与可能对该领域有所裨益，但目前 LLM 本身无法超过人类的表现水平。