While there have been extensive studies in code generation by large language
models (LLM), where benchmarks like HumanEval have been surpassed with an
impressive 96.3% success rate, these benchmarks predominantly judge a model's
performance on basic function-level code generation and lack the critical
thinking and concept of scope required of real-world scenarios such as solving
GitHub issues. This research introduces the application of the Tree of Thoughts
(ToT) language model reasoning framework for enhancing the decision-making and
problem-solving abilities of LLMs for this complex task. Compared to
traditional input-output (IO) prompting and Retrieval Augmented Generation
(RAG) techniques, ToT is designed to improve performance by facilitating a
structured exploration of multiple reasoning trajectories and enabling
self-assessment of potential solutions. We experimentally deploy ToT in
tackling a Github issue contained within an instance of the SWE-bench. However,
our results reveal that the ToT framework alone is not enough to give LLMs the
critical reasoning capabilities to outperform existing methods. In this paper
we analyze the potential causes of these shortcomings and identify key areas
for improvement such as deepening the thought process and introducing agentic
capabilities. The insights of this research are aimed at informing future
directions for refining the application of ToT and better harnessing the
potential of LLMs in real-world problem-solving scenarios.

本研究介绍了 Tree of Thoughts（ToT）语言模型推理框架的应用，以增强大型语言模型（LLM）在解决实际问题时的决策和问题解决能力。通过改进性能，促进多个推理轨迹的结构化探索和潜在解决方案的自我评估，ToT 框架被用于提高代码生成中复杂任务的性能。然而，研究结果显示，仅依靠 ToT 框架还不足以超越现有方法，并且需要加深思维过程和引入主体能力等方面进行改进。该研究的见解旨在为完善 ToT 的应用和更好地利用 LLMs 在实际问题解决场景中的潜力提供指导。

能用思维树解决 Github 问题吗？

Can Github issues be solved with Tree Of Thoughts?

Researchers have made significant progress in automating the software
development process in the past decades. Automated techniques for issue
summarization, bug reproduction, fault localization, and program repair have
been built to ease the workload of developers. Recent progress in Large
Language Models (LLMs) has significantly impacted the development process,
where developers can use LLM-based programming assistants to achieve automated
coding. Nevertheless software engineering involves the process of program
improvement apart from coding, specifically to enable software maintenance
(e.g. program repair to fix bugs) and software evolution (e.g. feature
additions). In this paper, we propose an automated approach for solving Github
issues to autonomously achieve program improvement. In our approach called
AutoCodeRover, LLMs are combined with sophisticated code search capabilities,
ultimately leading to a program modification or patch. In contrast to recent
LLM agent approaches from AI researchers and practitioners, our outlook is more
software engineering oriented. We work on a program representation (abstract
syntax tree) as opposed to viewing a software project as a mere collection of
files. Our code search exploits the program structure in the form of
classes/methods to enhance LLM's understanding of the issue's root cause, and
effectively retrieve a context via iterative search. The use of spectrum based
fault localization using tests, further sharpens the context. Experiments on
the recently proposed SWE-bench-lite which consists of 300 real-life Github
issues involving bug fixing and feature additions show increased efficacy
(resolving more than 20% on SWE-bench-lite), as compared to recent efforts from
the AI community. We posit that our workflow enables autonomous software
engineering, where, in future, auto-generated code from LLMs can be
autonomously improved.

提出了一种自动化的方法 AutoCodeRover，将 LLMs 与复杂的代码搜索能力相结合，从而实现对 Github 问题的自主解决，进而实现程序改进。