Large Language Models (LLMs) often suffer from overconfidence during
inference, particularly when adapted to downstream domain-specific tasks with
limited data. Previous work addresses this issue by employing approximate
Bayesian estimation after the LLMs are trained, enabling them to quantify
uncertainty. However, such post-training approaches' performance is severely
limited by the parameters learned during training. In this paper, we go beyond
post-training Bayesianization and propose Bayesian Low-Rank Adaptation by
Backpropagation (BLoB), an algorithm that continuously and jointly adjusts both
the mean and covariance of LLM parameters throughout the whole fine-tuning
process. Our empirical results verify the effectiveness of BLoB in terms of
generalization and uncertainty estimation, when evaluated on both
in-distribution and out-of-distribution data.

大语言模型（LLMs）在推断过程中常常过于自信，尤其是当它们适应具有有限数据的下游领域特定任务时。本文通过在 LLMs 训练后采用近似贝叶斯估计的方法来解决这个问题，从而使其能够量化不确定性。然而，这种训练后方法的性能受训练期间学习的参数严重限制。在本文中，我们超越了训练后的贝叶斯化，并提出了一种名为贝叶斯低秩适应的反向传播（BLoB）的算法，该算法在整个微调过程中持续和联合调整 LLM 参数的均值和协方差。我们的实证结果验证了 BLoB 在广义化与不确定性估计方面的有效性，同时在分布内和分布外数据上进行评估。