We investigate the training of sparse layers that use different parameters
for different inputs based on hashing in large Transformer models.
Specifically, we modify the feedforward layer to hash to different sets of
weights depending on the current token, over all tokens in the sequence. We
show that this procedure either outperforms or is competitive with
learning-to-route mixture-of-expert methods such as Switch Transformers and
BASE Layers, while requiring no routing parameters or extra terms in the
objective function such as a load balancing loss, and no sophisticated
assignment algorithm. We study the performance of different hashing techniques,
hash sizes and input features, and show that balanced and random hashes focused
on the most local features work best, compared to either learning clusters or
using longer-range context. We show our approach works well both on large
language modeling and dialogue tasks, and on downstream fine-tuning tasks.

本研究通过使用哈希方法对大型 Transformer 模型中使用不同输入参数的稀疏层进行了训练，通过修改前馈层对序列中的当前标记进行哈希以便将其哈希到不同的权重集中，进一步证明此方法可以在不需要额外路由参数或负载平衡损失等的情况下超越或与 Switch Transformers 和 BASE Layers 等学习路由混合专家方法竞争，同时研究了不同的哈希技术、哈希大小和输入特征的表现，并表明了本研究的方法可以在大规模语言建模和对话任务、下游微调任务中表现出色。