Advances in language modeling have paved the way for novel human-AI
co-writing experiences. This paper explores how varying levels of scaffolding
from large language models (LLMs) shape the co-writing process. Employing a
within-subjects field experiment with a Latin square design, we asked
participants (N=131) to respond to argumentative writing prompts under three
randomly sequenced conditions: no AI assistance (control), next-sentence
suggestions (low scaffolding), and next-paragraph suggestions (high
scaffolding). Our findings reveal a U-shaped impact of scaffolding on writing
quality and productivity (words/time). While low scaffolding did not
significantly improve writing quality or productivity, high scaffolding led to
significant improvements, especially benefiting non-regular writers and less
tech-savvy users. No significant cognitive burden was observed while using the
scaffolded writing tools, but a moderate decrease in text ownership and
satisfaction was noted. Our results have broad implications for the design of
AI-powered writing tools, including the need for personalized scaffolding
mechanisms.

大语言模型的不同程度脚手架支持对协作写作过程的影响进行了探讨。在一个拉丁方设计的实地实验中，共有 131 位参与者在无 AI 辅助（对照组）、下一句建议（低脚手架）和下一段建议（高脚手架）三种随机顺序的条件下回答辩论写作问题。研究发现，脚手架对写作质量和生产率（单位时间内的字数）的影响呈 U 型曲线。低脚手架并没有显著改善写作质量或生产率，而高脚手架则显著改善，特别是对于非常规写作者和对技术不熟悉的用户有益。使用脚手架写作工具时没有观察到明显的认知负荷，但注意到了对文本所有权和满意度的适度减少。我们的研究对于设计 AI 辅助写作工具具有广泛的意义，包括个性化脚手架机制的需求。