Humans regularly engage in analogical thinking, relating personal experiences
to current situations ($X$ is analogous to $Y$ because of $Z$). Analogical
thinking allows humans to solve problems in creative ways, grasp difficult
concepts, and articulate ideas more effectively. Can language models (LMs) do
the same? To answer this question, we propose ANALOBENCH, a benchmark to
determine analogical reasoning ability in LMs. Our benchmarking approach
focuses on aspects of this ability that are common among humans: (i) recalling
related experiences from a large amount of information, and (ii) applying
analogical reasoning to complex and lengthy scenarios. We test a broad
collection of proprietary models (e.g., GPT family, Claude V2) and open source
models such as LLaMA2. As in prior results, scaling up LMs results in some
performance boosts. Surprisingly, scale offers minimal gains when, (i)
analogies involve lengthy scenarios, or (ii) recalling relevant scenarios from
a large pool of information, a process analogous to finding a needle in a
haystack. We hope these observations encourage further research in this field.

用于评估语言模型中类比推理能力的基准模型 ANALOBENCH 可以从大量信息中回忆相关经验，并将类比推理应用于复杂和冗长的情景中。