The coverage and composition of the pretraining data corpus significantly
impacts the generalization ability of large language models. Conventionally,
the pretraining corpus is composed of various source domains (e.g. CommonCrawl,
Wikipedia, Github etc.) according to certain sampling probabilities (domain
weights). However, current methods lack a principled way to optimize domain
weights for ultimate goal for generalization. We propose DOmain reweighting
with Generalization Estimation (DoGE), where we reweigh the sampling
probability from each domain based on its contribution to the final
generalization objective assessed by a gradient-based generalization estimation
function. First, we train a small-scale proxy model with a min-max optimization
to obtain the reweighted domain weights. At each step, the domain weights are
updated to maximize the overall generalization gain by mirror descent. Finally
we use the obtained domain weights to train a larger scale full-size language
model. On SlimPajama-6B dataset, with universal generalization objective, DoGE
achieves better average perplexity and zero-shot reasoning accuracy. On
out-of-domain generalization tasks, DoGE reduces perplexity on the target
domain by a large margin. We further apply a parameter-selection scheme which
improves the efficiency of generalization estimation.

使用 DOmain reweighting with Generalization Estimation (DoGE) 方法，通过基于梯度的泛化估计函数来重新权衡每个领域的样本抽样概率，以达到最终泛化目标，从而提高大型语言模型的泛化能力。在 SlimPajama-6B 数据集上，DoGE 方法在普适泛化目标上获得了更好的平均困惑度和零样本推理准确性，同时在领域外泛化任务中，DoGE 显著降低了目标领域的困惑度。我们还应用了参数选择方案，提高了泛化估计的效率。

DoGE: 领域加权与泛化估计

DoGE: Domain Reweighting with Generalization Estimation

The worst-case training principle that minimizes the maximal adversarial
loss, also known as adversarial training (AT), has shown to be a
state-of-the-art approach for enhancing adversarial robustness. Nevertheless,
min-max optimization beyond the purpose of AT has not been rigorously explored
in the adversarial context. In this paper, we show how a general framework of
min-max optimization over multiple domains can be leveraged to advance the
design of different types of adversarial attacks. In particular, given a set of
risk sources, minimizing the worst-case attack loss can be reformulated as a
min-max problem by introducing domain weights that are maximized over the
probability simplex of the domain set. We showcase this unified framework in
three attack generation problems -- attacking model ensembles, devising
universal perturbation under multiple inputs, and crafting attacks resilient to
data transformations. Extensive experiments demonstrate that our approach leads
to substantial attack improvement over the existing heuristic strategies as
well as robustness improvement over state-of-the-art defense methods trained to
be robust against multiple perturbation types. Furthermore, we find that the
self-adjusted domain weights learned from our min-max framework can provide a
holistic tool to explain the difficulty level of attack across domains. Code is
available at this https URL

本文利用多域最小最大优化的一般性框架，推进了不同类型的对抗性攻击设计，并展示了该统一框架在攻击目标函数，抵御多输入下的统一扰动，生成攻击和数据变换下的对抗攻击等方面的优异性能和整体性工具的作用。