The rapid advancements in large language models (LLMs) have significantly improved their ability to generate natural language, making texts generated by LLMs increasingly indistinguishable from human-written texts. Recent research has predominantly focused on using LLMs to classify text as either human-written or machine-generated. In our study, we adopt a different approach by profiling texts spanning four domains based on 250 distinct linguistic features. We select the M4 dataset from the Subtask B of SemEval 2024 Task 8. We automatically calculate various linguistic features with the LFTK tool and additionally measure the average syntactic depth, semantic similarity, and emotional content for each document. We then apply a two-dimensional PCA reduction to all the calculated features. Our analyses reveal significant differences between human-written texts and those generated by LLMs, particularly in the variability of these features, which we find to be considerably higher in human-written texts. This discrepancy is especially evident in text genres with less rigid linguistic style constraints. Our findings indicate that humans write texts that are less cognitively demanding, with higher semantic content, and richer emotional content compared to texts generated by LLMs. These insights underscore the need for incorporating meaningful linguistic features to enhance the understanding of textual outputs of LLMs.

本研究针对大型语言模型（LLMs）生成文本的特征与人类写作之间的差异进行了分析。通过对250个不同语言特征的分析，我们发现人类文本的变异性显著高于机器生成文本，尤其在对语言风格约束较少的文本类型中更为明显。研究表明，人类创作的文本在认知需求、语义内容和情感表达上具有更丰富的特征，强调了在理解LLM输出时需要引入有意义的语言特征。

人类变异性与机器一致性：对人类与大型语言模型生成文本的语言分析