Social media bot detection has always been an arms race between advancements in machine learning bot detectors and adversarial bot strategies to evade detection. In this work, we bring the arms race to the next level by investigating the opportunities and risks of state-of-the-art large language models (LLMs) in social bot detection. To investigate the opportunities, we design novel LLM-based bot detectors by proposing a mixture-of-heterogeneous-experts framework to divide and conquer diverse user information modalities. To illuminate the risks, we explore the possibility of LLM-guided manipulation of user textual and structured information to evade detection. Extensive experiments with three LLMs on two datasets demonstrate that instruction tuning on merely 1,000 annotated examples produces specialized LLMs that outperform state-of-the-art baselines by up to 9.1% on both datasets, while LLM-guided manipulation strategies could significantly bring down the performance of existing bot detectors by up to 29.6% and harm the calibration and reliability of bot detection systems.

社交媒体机器人检测一直是机器学习机器人检测器和对抗机器人策略之间的一场军备竞赛。本研究将这场竞赛提升到了一个新的水平，通过研究最先进的大型语言模型（LLMs）在社交机器人检测中的机会和风险，设计了基于LLM的机器人检测器，并探索了LLM引导的操纵用户文本和结构化信息来逃避检测的可能性。实验结果表明，仅仅在1000个注释示例上进行的指令调优可以产生专门的LLMs，它们在两个数据集上的表现比最先进的基线方法提高了高达9.1%，而LLM引导的操纵策略可以将现有的机器人检测器的性能显著降低高达29.6%，并损害机器人检测系统的校准和可靠性。

大语言模型在社交媒体机器人检测中的机遇与风险