In this work, we propose FastCoT, a model-agnostic framework based on
parallel decoding without any further training of an auxiliary model or
modification to the LLM itself. FastCoT uses a size-varying context window
whose size changes with position to conduct parallel decoding and
auto-regressive decoding simultaneously, thus fully utilizing GPU computation
resources. In FastCoT, the parallel decoding part provides the LLM with a quick
glance of the future composed of approximate tokens, which could lead to faster
answers compared to regular autoregressive decoding used by causal
transformers. We also provide an implementation of parallel decoding within
LLM, which supports KV-cache generation and batch processing. Through extensive
experiments, we demonstrate that FastCoT saves inference time by nearly 20%
with only a negligible performance drop compared to the regular approach.
Additionally, we show that the context window size exhibits considerable
robustness for different tasks.

本文提出了 FastCoT，这是一个基于并行解码的模型无关框架，通过使用大小可变的上下文窗口来同时进行并行解码和自回归解码，从而充分利用 GPU 计算资源。通过广泛实验证明，FastCoT 相对于常规方法可以将推理时间缩短近 20％，而性能只有微不足道的下降。此外，研究还展示了上下文窗口大小对不同任务具有相当大的鲁棒性。