There has been a growing effort to replace hand extraction of data from
research papers with automated data extraction based on natural language
processing (NLP), language models (LMs), and recently, large language models
(LLMs). Although these methods enable efficient extraction of data from large
sets of research papers, they require a significant amount of up-front effort,
expertise, and coding. In this work we propose the ChatExtract method that can
fully automate very accurate data extraction with essentially no initial effort
or background using an advanced conversational LLM (or AI). ChatExtract
consists of a set of engineered prompts applied to a conversational LLM that
both identify sentences with data, extract data, and assure its correctness
through a series of follow-up questions. These follow-up questions address a
critical challenge associated with LLMs - their tendency to provide factually
inaccurate responses. ChatExtract can be applied with any conversational LLMs
and yields very high quality data extraction. In tests on materials data we
find precision and recall both over 90% from the best conversational LLMs,
likely rivaling or exceeding human accuracy in many cases. We demonstrate that
the exceptional performance is enabled by the information retention in a
conversational model combined with purposeful redundancy and introducing
uncertainty through follow-up prompts. These results suggest that approaches
similar to ChatExtract, due to their simplicity, transferability and accuracy
are likely to replace other methods of data extraction in the near future.

本研究提出了 ChatExtract 方法，该方法利用先进的会话型 LLM（或 AI）自动提取数据，并通过一系列后续问题确保其正确性。 ChatExtract 可用于任何会话型 LLMs，结果表明，由于其简单性，可转移性和准确性，类似 ChatExtract 的方法有望在不久的将来取代其他数据提取方法。

使用对话式语言模型和提示工程从研究论文中提取准确的材料数据 —— 以 ChatGPT 为例

Extracting Accurate Materials Data from Research Papers with Conversational Language Models and Prompt Engineering -- Example of ChatGPT

Conversational Agents (CAs) powered with deep language models (DLMs) have
shown tremendous promise in the domain of mental health. Prominently, the CAs
have been used to provide informational or therapeutic services to patients.
However, the utility of CAs to assist in mental health triaging has not been
explored in the existing work as it requires a controlled generation of
follow-up questions (FQs), which are often initiated and guided by the mental
health professionals (MHPs) in clinical settings. In the context of depression,
our experiments show that DLMs coupled with process knowledge in a mental
health questionnaire generate 12.54% and 9.37% better FQs based on similarity
and longest common subsequence matches to questions in the PHQ-9 dataset
respectively, when compared with DLMs without process knowledge support.
Despite coupling with process knowledge, we find that DLMs are still prone to
hallucination, i.e., generating redundant, irrelevant, and unsafe FQs. We
demonstrate the challenge of using existing datasets to train a DLM for
generating FQs that adhere to clinical process knowledge. To address this
limitation, we prepared an extended PHQ-9 based dataset, PRIMATE, in
collaboration with MHPs. PRIMATE contains annotations regarding whether a
particular question in the PHQ-9 dataset has already been answered in the
user's initial description of the mental health condition. We used PRIMATE to
train a DLM in a supervised setting to identify which of the PHQ-9 questions
can be answered directly from the user's post and which ones would require more
information from the user. Using performance analysis based on MCC scores, we
show that PRIMATE is appropriate for identifying questions in PHQ-9 that could
guide generative DLMs towards controlled FQ generation suitable for aiding
triaging. Dataset created as a part of this research:
this https URL

研究表明深度语言模型辅助的对话代理在心理健康方面具有很大潜力，用于提供信息性和治疗性服务，在抑郁症方面的实验显示控制生成问题对于精神保健评估是合适的，创造一个名为 PRIMATE 的数据集，用以培训深度语言模型。

利用过程知识学习自动化生成跟进问题以在 Reddit 帖子上进行抑郁症分流

Learning to Automate Follow-up Question Generation using Process Knowledge for Depression Triage on Reddit Posts

Conversational surveys, where an agent asks open-ended questions through
natural language interfaces, offer a new way to collect information from
people. A good follow-up question in a conversational survey prompts
high-quality information and delivers engaging experiences. However, generating
high-quality follow-up questions on the fly is a non-trivial task. The agent
needs to understand the diverse and complex participant responses, adhere to
the survey goal, and generate clear and coherent questions. In this study, we
propose a knowledge-driven follow-up question generation framework. The
framework combines a knowledge selection module to identify salient topics in
participants' responses and a generative model guided by selected knowledge
entity-relation pairs. To investigate the effectiveness of the proposed
framework, we build a new dataset for open-domain follow-up question generation
and present a new set of reference-free evaluation metrics based on Gricean
Maxim. Our experiments demonstrate that our framework outperforms a GPT-based
baseline in both objective evaluation and human-expert evaluation.

本研究提出一种基于知识驱动的 follow-up question 生成框架，结合知识选择模块和生成模型来提高开放域 follow-up question 的质量。实验证明，与 GPT 基线相比，该框架在客观评估和人类实验评估中都具有更好的性能。