Out-of-distribution (OOD) generalisation is challenging because it involves
not only learning from empirical data, but also deciding among various notions
of generalisation, e.g., optimising the average-case risk, worst-case risk, or
interpolations thereof. While this choice should in principle be made by the
model operator like medical doctors, this information might not always be
available at training time. The institutional separation between machine
learners and model operators leads to arbitrary commitments to specific
generalisation strategies by machine learners due to these deployment
uncertainties. We introduce the Imprecise Domain Generalisation framework to
mitigate this, featuring an imprecise risk optimisation that allows learners to
stay imprecise by optimising against a continuous spectrum of generalisation
strategies during training, and a model framework that allows operators to
specify their generalisation preference at deployment. Supported by both
theoretical and empirical evidence, our work showcases the benefits of
integrating imprecision into domain generalisation.

不确定域泛化是一个具有挑战性的问题，我们提出了模糊域泛化框架，通过不确定风险优化和模型框架的结合来解决此问题，并展示了将不确定性集成到域泛化中所带来的好处。

不精确学习的领域泛化

Domain Generalisation via Imprecise Learning

We introduce a causal regularisation extension to anchor regression (AR) for
improved out-of-distribution (OOD) generalisation. We present anchor-compatible
losses, aligning with the anchor framework to ensure robustness against
distribution shifts. Various multivariate analysis (MVA) algorithms, such as
(Orthonormalized) PLS, RRR, and MLR, fall within the anchor framework. We
observe that simple regularisation enhances robustness in OOD settings.
Estimators for selected algorithms are provided, showcasing consistency and
efficacy in synthetic and real-world climate science problems. The empirical
validation highlights the versatility of anchor regularisation, emphasizing its
compatibility with MVA approaches and its role in enhancing replicability while
guarding against distribution shifts. The extended AR framework advances causal
inference methodologies, addressing the need for reliable OOD generalisation.

我们通过引入因果正则化扩展到锚定回归（AR）来改善超出分布的广义。我们提出与锚定框架一致的锚定兼容损失，以确保对分布变化的鲁棒性。各种多变量分析（MVA）算法，如（正交化）PLS，RRR 和 MLR，都适用于锚定框架。我们观察到简单的正则化能增强在超出分布设置下的鲁棒性。我们提供了选择算法的估计器，展示了它们在合成和真实的气候科学问题中的一致性和功效。经验证实，锚定正则化的通用性凸显了其与 MVA 方法的兼容性，并在增强可复制性的同时防范了分布变化。扩展的 AR 框架推进了因果推断方法，解决了可靠的超出分布广义化的需求。

通过锚定多元分析提高泛化能力

Improving generalisation via anchor multivariate analysis

Large language models (LLMs) fine-tuned with reinforcement learning from
human feedback (RLHF) have been used in some of the most widely deployed AI
models to date, such as OpenAI's ChatGPT, Anthropic's Claude, or Meta's
LLaMA-2. While there has been significant work developing these methods, our
understanding of the benefits and downsides of each stage in RLHF is still
limited. To fill this gap, we present an extensive analysis of how each stage
of the process (i.e. supervised fine-tuning (SFT), reward modelling, and RLHF)
affects two key properties: out-of-distribution (OOD) generalisation and output
diversity. OOD generalisation is crucial given the wide range of real-world
scenarios in which these models are being used, while output diversity refers
to the model's ability to generate varied outputs and is important for a
variety of use cases. We perform our analysis across two base models on both
summarisation and instruction following tasks, the latter being highly relevant
for current LLM use cases. We find that RLHF generalises better than SFT to new
inputs, particularly as the distribution shift between train and test becomes
larger. However, RLHF significantly reduces output diversity compared to SFT
across a variety of measures, implying a tradeoff in current LLM fine-tuning
methods between generalisation and diversity. Our results provide guidance on
which fine-tuning method should be used depending on the application, and show
that more research is needed to improve the trade-off between generalisation
and diversity.

利用强化学习从人类反馈中对大型语言模型（LLMs）进行微调，该方法已在一些最广泛使用的人工智能模型中得到应用，如 OpenAI 的 ChatGPT、Anthropic 的 Claude 或 Meta 的 LLaMA-2。我们通过广泛的分析探讨了这个过程的每个阶段（即受监督的微调（SFT）、奖励建模和强化学习从人类反馈）对于两个关键属性的影响：越界分布的泛化和输出的多样性。我们发现，相对于受监督的微调，强化学习从人类反馈更好地泛化到新的输入，尤其是在训练和测试之间的分布差异越大时。然而，相比于受监督的微调，强化学习从人类反馈显著降低了输出的多样性，这意味着在当前 LLM 微调方法中存在泛化和多样性之间的权衡。我们的研究结果为特定应用提供了微调方法的指导，并表明有必要改进泛化和多样性之间的权衡。