The emergence of Large Language Models (LLMs) has great potential to reshape
the landscape of many social media platforms. While this can bring promising
opportunities, it also raises many threats, such as biases and privacy
concerns, and may contribute to the spread of propaganda by malicious actors.
We developed the "LLMs Among Us" experimental framework on top of the Mastodon
social media platform for bot and human participants to communicate without
knowing the ratio or nature of bot and human participants. We built 10 personas
with three different LLMs, GPT-4, LLama 2 Chat, and Claude. We conducted three
rounds of the experiment and surveyed participants after each round to measure
the ability of LLMs to pose as human participants without human detection. We
found that participants correctly identified the nature of other users in the
experiment only 42% of the time despite knowing the presence of both bots and
humans. We also found that the choice of persona had substantially more impact
on human perception than the choice of mainstream LLMs.

在 Mastodon 社交媒体平台上，我们使用了 “LLMs Among Us” 实验框架构建了 10 个角色来研究大型语言模型的潜力和威胁，发现参与者只有 42% 的准确率能识别出其他用户的真实性。此外，我们还发现角色的选择对于人的感知影响比主流大型语言模型的选择更大。