Recently, large language models (LLMs) have shown remarkable capabilities
including understanding context, engaging in logical reasoning, and generating
responses. However, this is achieved at the expense of stringent computational
and memory requirements, hindering their ability to effectively support long
input sequences. This survey provides an inclusive review of the recent
techniques and methods devised to extend the sequence length in LLMs, thereby
enhancing their capacity for long-context understanding. In particular, we
review and categorize a wide range of techniques including architectural
modifications, such as modified positional encoding and altered attention
mechanisms, which are designed to enhance the processing of longer sequences
while avoiding a proportional increase in computational requirements. The
diverse methodologies investigated in this study can be leveraged across
different phases of LLMs, i.e., training, fine-tuning and inference. This
enables LLMs to efficiently process extended sequences. The limitations of the
current methodologies is discussed in the last section along with the
suggestions for future research directions, underscoring the importance of
sequence length in the continued advancement of LLMs.

本文调查了扩展序列长度的技术和方法，包括架构修改和注意机制的改变等多种方法，并讨论了当前方法的局限性和未来研究方向建议，强调了序列长度对大型语言模型进一步发展的重要性。

超越极限：大型语言模型中扩展上下文长度的技术综述

Beyond the Limits: A Survey of Techniques to Extend the Context Length  in Large Language Models

The growing popularity of Vision Transformers as the go-to models for image
classification has led to an explosion of architectural modifications claiming
to be more efficient than the original ViT. However, a wide diversity of
experimental conditions prevents a fair comparison between all of them, based
solely on their reported results. To address this gap in comparability, we
conduct a comprehensive analysis of more than 30 models to evaluate the
efficiency of vision transformers and related architectures, considering
various performance metrics. Our benchmark provides a comparable baseline
across the landscape of efficiency-oriented transformers, unveiling a plethora
of surprising insights. For example, we discover that ViT is still Pareto
optimal across multiple efficiency metrics, despite the existence of several
alternative approaches claiming to be more efficient. Results also indicate
that hybrid attention-CNN models fare particularly well when it comes to low
inference memory and number of parameters, and also that it is better to scale
the model size, than the image size. Furthermore, we uncover a strong positive
correlation between the number of FLOPS and the training memory, which enables
the estimation of required VRAM from theoretical measurements alone.
Thanks to our holistic evaluation, this study offers valuable insights for
practitioners and researchers, facilitating informed decisions when selecting
models for specific applications. We publicly release our code and data at
this https URL

对视觉 Transformer 及相关架构的效率进行了综合分析，揭示了一系列有趣的见解，例如发现 ViT 在多个效率度量标准上仍然是最佳选择，同时低推理内存和参数数量时，混合注意力 - CNN 模型表现良好，模型大小的缩放比图像大小更为重要，FLOPS 与训练内存之间存在强正相关性。