We study the ability of state-of-the art models to answer constraint
satisfaction queries for information retrieval (e.g., 'a list of ice cream
shops in San Diego'). In the past, such queries were considered to be tasks
that could only be solved via web-search or knowledge bases. More recently,
large language models (LLMs) have demonstrated initial emergent abilities in
this task. However, many current retrieval benchmarks are either saturated or
do not measure constraint satisfaction. Motivated by rising concerns around
factual incorrectness and hallucinations of LLMs, we present KITAB, a new
dataset for measuring constraint satisfaction abilities of language models.
KITAB consists of book-related data across more than 600 authors and 13,000
queries, and also offers an associated dynamic data collection and constraint
verification approach for acquiring similar test data for other authors. Our
extended experiments on GPT4 and GPT3.5 characterize and decouple common
failure modes across dimensions such as information popularity, constraint
types, and context availability. Results show that in the absence of context,
models exhibit severe limitations as measured by irrelevant information,
factual errors, and incompleteness, many of which exacerbate as information
popularity decreases. While context availability mitigates irrelevant
information, it is not helpful for satisfying constraints, identifying
fundamental barriers to constraint satisfaction. We open source our
contributions to foster further research on improving constraint satisfaction
abilities of future models.

我们研究了目前最先进的模型在回答信息检索的约束满足查询（例如 “圣地亚哥的冰淇淋店列表”）方面的能力。我们介绍了 KITAB 数据集，它包括与 600 多位作者和 13000 多个查询相关的图书相关数据，并提供了相关的动态数据收集和约束验证方法，以获得其他作者的类似测试数据。结果表明，在缺乏上下文的情况下，模型在满足约束方面存在严重局限，包括与查询无关的信息、事实错误和不完整性等问题。尽管上下文的可用性可以减少与查询无关的信息，但它对满足约束并不有帮助，从而揭示出约束满足的根本障碍。我们开源我们的贡献，以促进未来模型在改善约束满足能力方面的进一步研究。