Transformer-based large language models (LLMs) typically have a limited
context window, resulting in significant performance degradation when
processing text beyond the length of the context window. Extensive studies have
been proposed to extend the context window and achieve length extrapolation of
LLMs, but there is still a lack of in-depth interpretation of these approaches.
In this study, we explore the positional information within and beyond the
context window for deciphering the underlying mechanism of LLMs. By using a
mean-based decomposition method, we disentangle positional vectors from hidden
states of LLMs and analyze their formation and effect on attention.
Furthermore, when texts exceed the context window, we analyze the change of
positional vectors in two settings, i.e., direct extrapolation and context
window extension. Based on our findings, we design two training-free context
window extension methods, positional vector replacement and attention window
extension. Experimental results show that our methods can effectively extend
the context window length.

通过分析位置向量对注意力的形成和影响，我们设计了两种无需训练的上下文窗口扩展方法，即位置向量替换和注意力窗口扩展。实验结果表明，我们的方法可以有效地扩展上下文窗口的长度。

通过分解位置向量探索大型语言模型的上下文窗口

Exploring Context Window of Large Language Models via Decomposed  Positional Vectors

Deep Implicit Function (DIF) has gained popularity as an efficient 3D shape
representation. To capture geometry details, current methods usually learn DIF
using local latent codes, which discretize the space into a regular 3D grid (or
octree) and store local codes in grid points (or octree nodes). Given a query
point, the local feature is computed by interpolating its neighboring local
codes with their positions. However, the local codes are constrained at
discrete and regular positions like grid points, which makes the code positions
difficult to be optimized and limits their representation ability. To solve
this problem, we propose to learn DIF with Dynamic Code Cloud, named DCC-DIF.
Our method explicitly associates local codes with learnable position vectors,
and the position vectors are continuous and can be dynamically optimized, which
improves the representation ability. In addition, we propose a novel code
position loss to optimize the code positions, which heuristically guides more
local codes to be distributed around complex geometric details. In contrast to
previous methods, our DCC-DIF represents 3D shapes more efficiently with a
small amount of local codes, and improves the reconstruction quality.
Experiments demonstrate that DCC-DIF achieves better performance over previous
methods. Code and data are available at this https URL

本文介绍了一种名为 DCC-DIF 的深度内隐函数学习方法，通过将本地编码与可学习的位置向量相对应，将 3D 形状表示为动态编码云，用于捕获几何细节，提高表示能力和重建质量。