Many studies have revealed that large language models (LLMs) exhibit uneven
awareness of different contextual positions.Their limited context awareness can
lead to overlooking critical information and subsequent task failures. While
several approaches have been proposed to enhance LLMs' context awareness,
achieving both effectiveness and efficiency remains challenging.In this paper,
for LLMs utilizing RoPE as position embeddings, we introduce a novel method
called ``Mixture of In-Context Experts'' (MoICE) to address this challenge.
MoICE comprises two key components: a router integrated into each attention
head within LLMs and a lightweight router-only training optimization strategy:
(1) MoICE views each RoPE angle as an `in-context' expert, demonstrated to be
capable of directing the attention of a head to specific contextual positions.
Consequently, each attention head flexibly processes tokens using multiple RoPE
angles dynamically selected by the router to attend to the needed positions.
This approach mitigates the risk of overlooking essential contextual
information. (2) The router-only training strategy entails freezing LLM
parameters and exclusively updating routers for only a few steps. When applied
to open-source LLMs including Llama and Mistral, MoICE surpasses prior methods
across multiple tasks on long context understanding and generation, all while
maintaining commendable inference efficiency.

本文提出了一种名为 “MoICE” 的新方法，通过在每个注意力头中引入路由器作为关键部件以及仅更新路由器进行训练的策略，来提高使用 RoPE 作为位置嵌入的 LLMs 的上下文感知能力和效率。在应用于 Llama 和 Mistral 等开源 LLMs 时，MoICE 在长上下文理解和生成的多个任务上超越了先前的方法，并保持了可观的推理效率。