Large Language Models (LLMs) have shown remarkable promise in their ability
to interact proficiently with humans. Subsequently, their potential use as
artificial confederates and surrogates in sociological experiments involving
conversation is an exciting prospect. But how viable is this idea? This paper
endeavors to test the limits of current-day LLMs with a pre-registered study
integrating real people with LLM agents acting as people. The study focuses on
debate-based opinion consensus formation in three environments: humans only,
agents and humans, and agents only. Our goal is to understand how LLM agents
influence humans, and how capable they are in debating like humans. We find
that LLMs can blend in and facilitate human productivity but are less
convincing in debate, with their behavior ultimately deviating from human's. We
elucidate these primary failings and anticipate that LLMs must evolve further
before being viable debaters.

当前的研究论文测试了大型语言模型（LLMs）的可行性，作为人工代理人参与社会学实验中的对话活动，揭示了 LLMs 在辩论中的限制，以及其对人类的影响和能力。