We develop new conformal inference methods for obtaining validity guarantees
on the output of large language models (LLMs). Prior work in conformal language
modeling identifies a subset of the text that satisfies a high-probability
guarantee of correctness. These methods work by filtering claims from the LLM's
original response if a scoring function evaluated on the claim fails to exceed
a threshold calibrated via split conformal prediction. Existing methods in this
area suffer from two deficiencies. First, the guarantee stated is not
conditionally valid. The trustworthiness of the filtering step may vary based
on the topic of the response. Second, because the scoring function is
imperfect, the filtering step can remove many valuable and accurate claims. We
address both of these challenges via two new conformal methods. First, we
generalize the conditional conformal procedure of Gibbs et al. (2023) in order
to adaptively issue weaker guarantees when they are required to preserve the
utility of the output. Second, we show how to systematically improve the
quality of the scoring function via a novel algorithm for differentiating
through the conditional conformal procedure. We demonstrate the efficacy of our
approach on both synthetic and real-world datasets.

我们开发了新的一致推断方法，用于获取大型语言模型 (LLMs) 输出的有效性保证。我们通过过滤控制函数的评估结果来确定出 LLM 响应中满足高概率正确性保证的文本子集。我们的方法解决了现有方法存在的两个问题，首先，所述保证并非条件有效；其次，因为评分函数不完善，过滤步骤可能会删除许多有价值和准确的声明，我们通过两种新的一致方法解决了这两个挑战。