Divergent thinking, the cognitive process of generating diverse solutions, is
a hallmark of human creativity and problem-solving. For machines, sampling
diverse solution trajectories in complex reasoning problems is crucial for
robust outcomes, data augmentation, and enhanced model generalization. Large
language models (LLMs) often struggle with generating high-quality, diverse
reasoning. While supervised fine-tuning helps with quality, it requires
extensive supervision data to capture the full diversity of solutions.
Alternatively, reinforcement learning methods like PPO aim to find limited
highest-reward solutions while neglecting the solution diversity, akin to
convergent thinking. To address these limitations, we propose Flow of Reasoning
(FoR) -- an efficient LLM training approach enabling diverse reasoning with
minimal data. FoR formulates multi-step LLM reasoning as a Markovian flow from
an initial state to terminal states. The formulation allows to adapt principled
GFlowNet approaches to train the LLM as a policy, which is able to sample
multiple reasoning paths with probabilities proportional to the unnormalized
reward. Empirical results show that, with limited training data (e.g., 15
examples), FoR can discover diverse high-quality solutions that excel greatly
beyond current state-of-the-art methods across three tasks, including embodied
reasoning (BlocksWorld), math puzzle solving (Game24), and logical reasoning
(PrOntoQA). Code is available at this https URL

通过 Flow of Reasoning（FoR）这一高效的 LLM 训练方法，能够在很少的训练数据的情况下，发现多样化高质量的解决方案，并且在三项任务中（包括具身化推理、数学谜题解决和逻辑推理）显著超越当前最先进的方法。

推理的流程：具有发散思维的 LLM 策略的高效训练

Flow of Reasoning: Efficient Training of LLM Policy with Divergent  Thinking

The recent surge in the capabilities of Large Language Models (LLMs) has led
to claims that they are approaching a level of creativity akin to human
capabilities. This idea has sparked a blend of excitement and apprehension.
However, a critical piece that has been missing in this discourse is a
systematic evaluation of LLM creativity, particularly in comparison to human
divergent thinking. To bridge this gap, we leverage recent advances in
creativity science to build a framework for in-depth analysis of divergent
creativity in both state-of-the-art LLMs and a substantial dataset of 100,000
humans. We found evidence suggesting that LLMs can indeed surpass human
capabilities in specific creative tasks such as divergent association and
creative writing. Our quantitative benchmarking framework opens up new paths
for the development of more creative LLMs, but it also encourages more granular
inquiries into the distinctive elements that constitute human inventive thought
processes, compared to those that can be artificially generated.

最新研究表明，大型语言模型（LLMs）在创造性方面可以超越人类的能力，特别是在发散性思维和创造性写作等特定创造性任务方面，这项研究使用创造力科学的最新进展构建了一个评估 LLMs 创造力的深入分析框架与一个包含 10 万人的实验数据集进行对比，既为更具创造力的 LLMs 的发展开辟了新的路径，也促进了对由人类独特的创新思维过程构成的与那些可以人工生成的元素的更细致的研究。