We develop new conformal inference methods for obtaining validity guarantees
on the output of large language models (LLMs). Prior work in conformal language
modeling identifies a subset of the text that satisfies a high-probability
guarantee of correctness. These methods work by filtering claims from the LLM's
original response if a scoring function evaluated on the claim fails to exceed
a threshold calibrated via split conformal prediction. Existing methods in this
area suffer from two deficiencies. First, the guarantee stated is not
conditionally valid. The trustworthiness of the filtering step may vary based
on the topic of the response. Second, because the scoring function is
imperfect, the filtering step can remove many valuable and accurate claims. We
address both of these challenges via two new conformal methods. First, we
generalize the conditional conformal procedure of Gibbs et al. (2023) in order
to adaptively issue weaker guarantees when they are required to preserve the
utility of the output. Second, we show how to systematically improve the
quality of the scoring function via a novel algorithm for differentiating
through the conditional conformal procedure. We demonstrate the efficacy of our
approach on both synthetic and real-world datasets.

我们开发了新的一致推断方法，用于获取大型语言模型 (LLMs) 输出的有效性保证。我们通过过滤控制函数的评估结果来确定出 LLM 响应中满足高概率正确性保证的文本子集。我们的方法解决了现有方法存在的两个问题，首先，所述保证并非条件有效；其次，因为评分函数不完善，过滤步骤可能会删除许多有价值和准确的声明，我们通过两种新的一致方法解决了这两个挑战。

通过增强的符合预测方法验证大型语言模型的有效性

Large language model validity via enhanced conformal prediction methods

Score-based explainable machine-learning techniques are often used to
understand the logic behind black-box models. However, such explanation
techniques are often computationally expensive, which limits their application
in time-critical contexts. Therefore, we propose and investigate the use of
computationally less costly regression models for approximating the output of
score-based explanation techniques, such as SHAP. Moreover, validity guarantees
for the approximated values are provided by the employed inductive conformal
prediction framework. We propose several non-conformity measures designed to
take the difficulty of approximating the explanations into account while
keeping the computational cost low. We present results from a large-scale
empirical investigation, in which the approximate explanations generated by our
proposed models are evaluated with respect to efficiency (interval size). The
results indicate that the proposed method can significantly improve execution
time compared to the fast version of SHAP, TreeSHAP. The results also suggest
that the proposed method can produce tight intervals, while providing validity
guarantees. Moreover, the proposed approach allows for comparing explanations
of different approximation methods and selecting a method based on how
informative (tight) are the predicted intervals.

我们提出并研究了使用计算成本较低的回归模型来逼近诸如 SHAP 之类的基于分数解释技术的输出，通过采用归纳符合预测框架提供了对逼近值的有效性保证。我们提出了几种非一致性度量方法，旨在考虑到解释逼近的困难程度同时保持计算成本的低廉。通过大规模实证研究的结果表明，我们提出的模型生成的近似解释在效率（区间大小）方面得到了评估。结果表明，与快速版本的 SHAP TreeSHAP 相比，所提出的方法可以显著提高执行时间。结果还表明，所提出的方法可以产生紧密的区间，同时提供有效性保证。此外，所提出的方法允许比较不同逼近方法的解释，并根据预测区间的信息量（紧密程度）选择一种方法。