Self-alignment is an effective way to reduce the cost of human annotation
while ensuring promising model capability. However, most current methods
complete the data collection and training steps in a single round, which may
overlook the continuously improving ability of self-aligned models. This gives
rise to a key query: What if we do multi-time bootstrapping self-alignment?
Does this strategy enhance model performance or lead to rapid degradation? In
this paper, our pioneering exploration delves into the impact of bootstrapping
self-alignment on large language models. Our findings reveal that bootstrapping
self-alignment markedly surpasses the single-round approach, by guaranteeing
data diversity from in-context learning. To further exploit the capabilities of
bootstrapping, we investigate and adjust the training order of data, which
yields improved performance of the model. Drawing on these findings, we propose
Step-On-Feet Tuning (SOFT) which leverages model's continuously enhanced
few-shot ability to boost zero or one-shot performance. Based on easy-to-hard
training recipe, we propose SOFT+ which further boost self-alignment's
performance. Our experiments demonstrate the efficiency of SOFT (SOFT+) across
various classification and generation tasks, highlighting the potential of
bootstrapping self-alignment on continually enhancing model alignment
performance.

自校准是降低人工标注成本并确保模型能力的有效方法。本文探讨了多次引导自校准对大型语言模型的影响，发现多次引导自校准明显优于单轮引导，通过在上下文学习中保证数据多样性。基于这些发现，提出了 Step-On-Feet Tuning（SOFT）用于提升零次或一次试验效果。SOFT + 进一步提升了自校准的性能，实验证明了多任务分类和生成任务上 SOFT（SOFT+）的效率，突显了多次引导自校准在不断增强模型对齐性能方面的潜力。