Numerous AI-assisted scholarly applications have been developed to aid
different stages of the research process. We present an analysis of AI-assisted
scholarly writing generated with ScholaCite, a tool we built that is designed
for organizing literature and composing Related Work sections for academic
papers. Our evaluation method focuses on the analysis of citation graphs to
assess the structural complexity and inter-connectedness of citations in texts
and involves a three-way comparison between (1) original human-written texts,
(2) purely GPT-generated texts, and (3) human-AI collaborative texts. We find
that GPT-4 can generate reasonable coarse-grained citation groupings to support
human users in brainstorming, but fails to perform detailed synthesis of
related works without human intervention. We suggest that future writing
assistant tools should not be used to draft text independently of the human
author.

通过分析引文图，我们评估了人工智能辅助学术写作，并发现 GPT-4 在支持人类用户的构思方面能够生成合理的粗粒度引文组合，但在没有人为干预的情况下无法进行详细的相关工作综合。因此，我们建议未来的写作助手工具不应独立撰写文本。