Effectively handling instructions with extremely long context remains a
challenge for Large Language Models (LLMs), typically necessitating
high-quality long data and substantial computational resources. This paper
introduces Step-Skipping Alignment (SkipAlign), a new technique designed to
enhance the long-context capabilities of LLMs in the phase of alignment without
the need for additional efforts beyond training with original data length.
SkipAlign is developed on the premise that long-range dependencies are
fundamental to enhancing an LLM's capacity of long context. Departing from
merely expanding the length of input samples, SkipAlign synthesizes long-range
dependencies from the aspect of positions indices. This is achieved by the
strategic insertion of skipped positions within instruction-following samples,
which utilizes the semantic structure of the data to effectively expand the
context. Through extensive experiments on base models with a variety of context
window sizes, SkipAlign demonstrates its effectiveness across a spectrum of
long-context tasks. Particularly noteworthy is that with a careful selection of
the base model and alignment datasets, SkipAlign with only 6B parameters
achieves it's best performance and comparable with strong baselines like
GPT-3.5-Turbo-16K on LongBench.

该研究论文介绍了一种名为 Step-Skipping Alignment 的技术，为大型语言模型（LLMs）增强了长上下文能力，通过在指令 - 跟随样本中策略性地插入跳过的位置来合成长程依赖，从而有效扩展上下文，并通过在不同上下文窗口大小的基础模型上进行广泛实验来验证其有效性。