There has been a growing effort to replace hand extraction of data from
research papers with automated data extraction based on natural language
processing (NLP), language models (LMs), and recently, large language models
(LLMs). Although these methods enable efficient extraction of data from large
sets of research papers, they require a significant amount of up-front effort,
expertise, and coding. In this work we propose the ChatExtract method that can
fully automate very accurate data extraction with essentially no initial effort
or background using an advanced conversational LLM (or AI). ChatExtract
consists of a set of engineered prompts applied to a conversational LLM that
both identify sentences with data, extract data, and assure its correctness
through a series of follow-up questions. These follow-up questions address a
critical challenge associated with LLMs - their tendency to provide factually
inaccurate responses. ChatExtract can be applied with any conversational LLMs
and yields very high quality data extraction. In tests on materials data we
find precision and recall both over 90% from the best conversational LLMs,
likely rivaling or exceeding human accuracy in many cases. We demonstrate that
the exceptional performance is enabled by the information retention in a
conversational model combined with purposeful redundancy and introducing
uncertainty through follow-up prompts. These results suggest that approaches
similar to ChatExtract, due to their simplicity, transferability and accuracy
are likely to replace other methods of data extraction in the near future.

本研究提出了 ChatExtract 方法，该方法利用先进的会话型 LLM（或 AI）自动提取数据，并通过一系列后续问题确保其正确性。 ChatExtract 可用于任何会话型 LLMs，结果表明，由于其简单性，可转移性和准确性，类似 ChatExtract 的方法有望在不久的将来取代其他数据提取方法。