The rapid development of large language models (LLMs) in recent years has
largely focused on English, resulting in models that respond exclusively in
English. To adapt these models to other languages, continual pre-training (CP)
is often employed, followed by supervised fine-tuning (SFT) to maintain
conversational abilities. However, CP and SFT can reduce a model's ability to
filter harmful content. We propose Instruction Continual Pre-training (InsCP),
which integrates instruction tags into the CP process to prevent loss of
conversational proficiency while acquiring new languages. Our experiments
demonstrate that InsCP retains conversational and Reinforcement Learning from
Human Feedback (RLHF) abilities. Empirical evaluations on language alignment,
reliability, and knowledge benchmarks confirm the efficacy of InsCP. Notably,
this approach requires only 0.1 billion tokens of high-quality
instruction-following data, thereby reducing resource consumption.

通过 Instruction Continual Pre-training (InsCP) 的方法，可以在维持对话能力的同时，将大型语言模型（LLMs）调整为适应其他语言，从而避免对有害内容过滤的能力下降，且只需要 0.1 十亿个高质量的指令遵循数据的资源消耗相对较低。