Recent neural language models have taken a significant step forward in
producing remarkably controllable, fluent, and grammatical text. Although
studies have found that AI-generated text is not distinguishable from
human-written text for crowd-sourcing workers, there still exist errors in
AI-generated text which are even subtler and harder to spot. We primarily focus
on the scenario in which scientific AI writing assistant is deeply involved.
First, we construct a feature description framework to distinguish between
AI-generated text and human-written text from syntax, semantics, and pragmatics
based on the human evaluation. Then we utilize the features, i.e., writing
style, coherence, consistency, and argument logistics, from the proposed
framework to analyze two types of content. Finally, we adopt several publicly
available methods to investigate the gap of between AI-generated scientific
text and human-written scientific text by AI-generated scientific text
detection models. The results suggest that while AI has the potential to
generate scientific content that is as accurate as human-written content, there
is still a gap in terms of depth and overall quality. The AI-generated
scientific content is more likely to contain errors in factual issues. We find
that there exists a "writing style" gap between AI-generated scientific text
and human-written scientific text. Based on the analysis result, we summarize a
series of model-agnostic and distribution-agnostic features for detection tasks
in other domains. Findings in this paper contribute to guiding the optimization
of AI models to produce high-quality content and addressing related ethical and
security concerns.

本文针对科学文献中 AI 生成的文本与人类编写的文本之间的差距进行了探讨，提出了一个通过语法、语义和语用来区分 AI 文本的框架，进而将提取到的特征用于分析两种不同类型的内容，发现 AI-generate 的科学文本在深度和总体质量方面还有待提高，并存在事实性问题等方面的差距，同时我们发现 AI-generate 的科学文本和人类编写的科学文本之间存在 “写作风格” 方面的差距，从而提出一些模型和分布无关的特征来用于其他领域的检测任务，这些研究结果有助于指导 AI 模型的优化，以产生高质量的文本，同时也有助于解决相关的伦理和安全问题。