Divergent thinking, the cognitive process of generating diverse solutions, is
a hallmark of human creativity and problem-solving. For machines, sampling
diverse solution trajectories in complex reasoning problems is crucial for
robust outcomes, data augmentation, and enhanced model generalization. Large
language models (LLMs) often struggle with generating high-quality, diverse
reasoning. While supervised fine-tuning helps with quality, it requires
extensive supervision data to capture the full diversity of solutions.
Alternatively, reinforcement learning methods like PPO aim to find limited
highest-reward solutions while neglecting the solution diversity, akin to
convergent thinking. To address these limitations, we propose Flow of Reasoning
(FoR) -- an efficient LLM training approach enabling diverse reasoning with
minimal data. FoR formulates multi-step LLM reasoning as a Markovian flow from
an initial state to terminal states. The formulation allows to adapt principled
GFlowNet approaches to train the LLM as a policy, which is able to sample
multiple reasoning paths with probabilities proportional to the unnormalized
reward. Empirical results show that, with limited training data (e.g., 15
examples), FoR can discover diverse high-quality solutions that excel greatly
beyond current state-of-the-art methods across three tasks, including embodied
reasoning (BlocksWorld), math puzzle solving (Game24), and logical reasoning
(PrOntoQA). Code is available at this https URL

通过 Flow of Reasoning（FoR）这一高效的 LLM 训练方法，能够在很少的训练数据的情况下，发现多样化高质量的解决方案，并且在三项任务中（包括具身化推理、数学谜题解决和逻辑推理）显著超越当前最先进的方法。