We explore the creative problem-solving capabilities of modern large language
models (LLMs) in a constrained setting. The setting requires circumventing a
cognitive bias known in psychology as ''functional fixedness'' to use familiar
objects in innovative or unconventional ways. To this end, we create MacGyver,
an automatically generated dataset consisting of 1,600 real-world problems that
deliberately trigger functional fixedness and require thinking
'out-of-the-box'. We then present our collection of problems to both LLMs and
humans to compare and contrast their problem-solving abilities. We show that
MacGyver is challenging for both groups, but in unique and complementary ways.
For example, humans typically excel in solving problems that they are familiar
with but may struggle with tasks requiring domain-specific knowledge, leading
to a higher variance. On the other hand, LLMs, being exposed to a variety of
highly specialized knowledge, attempt broader problems but are prone to
overconfidence and propose actions that are physically infeasible or
inefficient. We also provide a detailed error analysis of LLMs, and demonstrate
the potential of enhancing their problem-solving ability with novel prompting
techniques such as iterative step-wise reflection and divergent-convergent
thinking. This work provides insight into the creative problem-solving
capabilities of humans and AI and illustrates how psychological paradigms can
be extended into large-scale tasks for comparing humans and machines.

我们研究了现代大型语言模型（LLMs）在一个受限制的环境中的创造性问题解决能力。为了达到这个目的，我们创建了一个名为 'MacGyver' 的自动生成数据集，包含 1,600 个故意触发功能固定性并需要 ' 跳出框架 ' 思考的现实世界问题。我们将问题集呈现给 LLMs 和人类，并比较两者的问题解决能力。我们展示了对于两组来说，MacGyver 都是具有挑战性的，但方式是独特且互补的。我们还提供了 LLMs 的详细错误分析，并展示了通过新颖的提示技术（如迭代的逐步反思和发散 - 收敛思维）来增强它们的问题解决能力的潜力。这项工作揭示了人类和人工智能的创造性问题解决能力，并说明了如何将心理学范式扩展到比较人类和机器的大规模任务中。