Recent work demonstrated Transformers' ability to efficiently copy strings of
exponential sizes, distinguishing them from other architectures. We present the
Causal Relation Network (CausalRN), an all-MLP sequence modeling architecture
that can match Transformers on the copying task. Extending Relation Networks
(RNs), we implemented key innovations to support autoregressive sequence
modeling while maintaining computational feasibility. We discovered that
exponentially-activated RNs are reducible to linear time complexity, and
pre-activation normalization induces an infinitely growing memory pool, similar
to a KV cache. In ablation study, we found both exponential activation and
pre-activation normalization are indispensable for Transformer-level copying.
Our findings provide new insights into what actually constitutes strong
in-context retrieval.

Causal Relation Network (CausalRN) is an MLP sequence modeling architecture that can match Transformers on the copying task, employing key innovations such as exponentially-activated RNs, pre-activation normalization, and providing insights into strong in-context retrieval.

一个优秀的复制能力的全 MLP 序列建模架构

An All-MLP Sequence Modeling Architecture That Excels at Copying

Long-context language models (LCLMs) have the potential to revolutionize our
approach to tasks traditionally reliant on external tools like retrieval
systems or databases. Leveraging LCLMs' ability to natively ingest and process
entire corpora of information offers numerous advantages. It enhances
user-friendliness by eliminating the need for specialized knowledge of tools,
provides robust end-to-end modeling that minimizes cascading errors in complex
pipelines, and allows for the application of sophisticated prompting techniques
across the entire system. To assess this paradigm shift, we introduce LOFT, a
benchmark of real-world tasks requiring context up to millions of tokens
designed to evaluate LCLMs' performance on in-context retrieval and reasoning.
Our findings reveal LCLMs' surprising ability to rival state-of-the-art
retrieval and RAG systems, despite never having been explicitly trained for
these tasks. However, LCLMs still face challenges in areas like compositional
reasoning that are required in SQL-like tasks. Notably, prompting strategies
significantly influence performance, emphasizing the need for continued
research as context lengths grow. Overall, LOFT provides a rigorous testing
ground for LCLMs, showcasing their potential to supplant existing paradigms and
tackle novel tasks as model capabilities scale.

基于长上下文语言模型（LCLMs）的 LOFT 基准评估了 LCLMs 在上下文检索和推理方面的性能，发现它们能与最先进的检索系统和 RAG 系统媲美，但在像 SQL 一样需要组合推理的领域仍存在挑战，提示着需要继续研究随着上下文长度增长时的提示策略的影响。LOFT 为 LCLMs 提供了严格的测试平台，展示了随着模型能力的扩展，它们取代现有范式并解决新任务的潜力。

长上下文语言模型是否能包含检索、RAG、SQL 等功能？

Can Long-Context Language Models Subsume Retrieval, RAG, SQL, and More?

Long-context large language models (LLMs) hold promise for tasks such as
question-answering (QA) over long documents, but they tend to miss important
information in the middle of context documents (arXiv:2307.03172v3). Here, we
introduce $\textit{R&R}$ -- a combination of two novel prompt-based methods
called $\textit{reprompting}$ and $\textit{in-context retrieval}$ (ICR) -- to
alleviate this effect in document-based QA. In reprompting, we repeat the
prompt instructions periodically throughout the context document to remind the
LLM of its original task. In ICR, rather than instructing the LLM to answer the
question directly, we instruct it to retrieve the top $k$ passage numbers most
relevant to the given question, which are then used as an abbreviated context
in a second QA prompt. We test R&R with GPT-4 Turbo and Claude-2.1 on documents
up to 80k tokens in length and observe a 16-point boost in QA accuracy on
average. Our further analysis suggests that R&R improves performance on long
document-based QA because it reduces the distance between relevant context and
the instructions. Finally, we show that compared to short-context chunkwise
methods, R&R enables the use of larger chunks that cost fewer LLM calls and
output tokens, while minimizing the drop in accuracy.

通过两种新型的基于提示的方法 reprompting 和 in-context retrieval（ICR）相结合的方式，提出 R&D-- 一种改进长文档中问题回答（QA）效果的方法。在实验证明，使用 R&R 可以提高 QA 准确性。