Long sequence modeling has gained broad interest as large language models (LLMs) continue to advance. Recent research has identified that a large portion of hidden states within the key-value caches of Transformer models can be discarded (also termed evicted) without affecting the perplexity performance in generating long sequences. However, we show that these methods, despite preserving perplexity performance, often drop information that is important for solving downstream tasks, a problem which we call information neglect. To address this issue, we introduce Chunked Instruction-aware State Eviction (CItruS), a novel modeling technique that integrates the attention preferences useful for a downstream task into the eviction process of hidden states. In addition, we design a method for chunked sequence processing to further improve efficiency. Our training-free method exhibits superior performance on long sequence comprehension and retrieval tasks over several strong baselines under the same memory budget, while preserving language modeling perplexity.

通过引入一种新的建模技术CItruS，在隐藏状态的清理过程中集成下游任务中有用的注意力偏好，以解决信息忽略的问题，同时设计了一种分块序列处理方法来提高效率，该方法在相同的内存预算下在长序列理解和检索任务上表现出优越性能。

CItruS: 分块指令感知的长序列建模状态驱逐