Multi-agent interactions between Large Language Model (LLM) agents have shown
major improvements on diverse reasoning tasks. However, these involve long
generations from multiple models across several rounds, making them expensive.
Moreover, these multi-agent approaches fail to provide a final, single model
for efficient inference. To address this, we introduce MAGDi, a new method for
structured distillation of the reasoning interactions between multiple LLMs
into smaller LMs. MAGDi teaches smaller models by representing multi-agent
interactions as graphs, augmenting a base student model with a graph encoder,
and distilling knowledge using three objective functions: next-token
prediction, a contrastive loss between correct and incorrect reasoning, and a
graph-based objective to model the interaction structure. Experiments on seven
widely-used commonsense and math reasoning benchmarks show that MAGDi improves
the reasoning capabilities of smaller models, outperforming several methods
that distill from a single teacher and multiple teachers. Moreover, MAGDi also
demonstrates an order of magnitude higher efficiency over its teachers. We
conduct extensive analyses to show that MAGDi (1) enhances the generalizability
to out-of-domain tasks, (2) scales positively with the size and strength of the
base student model, and (3) obtains larger improvements (via our multi-teacher
training) when applying self-consistency - an inference technique that relies
on model diversity.

通过多个大型语言模型之间的多智能体交互，结构化蒸馏途径（MAGDi）通过将多智能体交互表示为图形、通过图形编码器增强基础学生模型，并使用三个目标函数进行知识蒸馏，即下一个标记预测、正确和错误推理之间的对比损失以及基于图形的目标函数。对七个广泛使用的常识和数学推理基准进行的实验证明，MAGDi 提高了较小模型的推理能力，表现优于从单个教师和多个教师进行蒸馏的几种方法。此外，MAGDi 还比其教师高出数量级的效率。我们进行了广泛的分析，显示 MAGDi（1）增强了对域外任务的泛化能力，（2）与基础学生模型的大小和强度正相关，（3）在应用自一致性的多教师训练方面取得了更大的改进，后者是一种依赖于模型多样性的推理技术。