Learned reweighting (LRW) approaches to supervised learning use an
optimization criterion to assign weights for training instances, in order to
maximize performance on a representative validation dataset. We pose and
formalize the problem of optimized selection of the validation set used in LRW
training, to improve classifier generalization. In particular, we show that
using hard-to-classify instances in the validation set has both a theoretical
connection to, and strong empirical evidence of generalization. We provide an
efficient algorithm for training this meta-optimized model, as well as a simple
train-twice heuristic for careful comparative study. We demonstrate that LRW
with easy validation data performs consistently worse than LRW with hard
validation data, establishing the validity of our meta-optimization problem.
Our proposed algorithm outperforms a wide range of baselines on a range of
datasets and domain shift challenges (Imagenet-1K, CIFAR-100, Clothing-1M,
CAMELYON, WILDS, etc.), with ~1% gains using VIT-B on Imagenet. We also show
that using naturally hard examples for validation (Imagenet-R / Imagenet-A) in
LRW training for Imagenet improves performance on both clean and naturally hard
test instances by 1-2%. Secondary analyses show that using hard validation data
in an LRW framework improves margins on test data, hinting at the mechanism
underlying our empirical gains. We believe this work opens up new research
directions for the meta-optimization of meta-learning in a supervised learning
context.

学习重加权（LRW）方法用优化准则分配训练实例权重，以在代表性验证数据集上最大化性能。我们提出并形式化了在 LRW 训练中优化选择验证集的问题，以提高分类器的泛化能力。使用难以分类的实例作为验证集，在理论上与实证的有效性存在强烈的关联。我们提供了一种有效的算法来训练这种元优化模型，并提出了一种简单的两次训练启发式方法进行比较研究。我们证明相对于易用验证数据的 LRW，具有难度验证数据的 LRW 表现一致更差，从而验证了我们元优化问题的有效性。我们的算法在一系列数据集和域偏移挑战（Imagenet-1K，CIFAR-100，Clothing-1M，CAMELYON，WILDS 等）上优于广泛的基线，通过在 Imagenet 上使用 VIT-B 获得了约 1％的增益。我们还展示了在 LRW 的训练中使用自然难的示例（Imagenet-R/Imagenet-A）能够在干净和自然难的测试实例上分别提高 1-2％的性能。次要分析显示，在 LRW 框架中使用难度验证数据可以提高测试数据的边界，暗示了我们实证增益背后的机制。我们认为这项工作为监督学习环境中元学习的元优化开辟了新的研究方向。

通过在难样本上进行元学习以提高泛化能力

Improving Generalization via Meta-Learning on Hard Samples

Mutual Information (MI) based feature selection makes use of MI to evaluate
each feature and eventually shortlists a relevant feature subset, in order to
address issues associated with high-dimensional datasets. Despite the
effectiveness of MI in feature selection, we notice that many state-of-the-art
algorithms disregard the so-called unique relevance (UR) of features, and
arrive at a suboptimal selected feature subset which contains a non-negligible
number of redundant features. We point out that the heart of the problem is
that all these MIBFS algorithms follow the criterion of Maximize Relevance with
Minimum Redundancy (MRwMR), which does not explicitly target UR. This motivates
us to augment the existing criterion with the objective of boosting unique
relevance (BUR), leading to a new criterion called MRwMR-BUR. Depending on the
task being addressed, MRwMR-BUR has two variants, termed MRwMR-BUR-KSG and
MRwMR-BUR-CLF, which estimate UR differently. MRwMR-BUR-KSG estimates UR via a
nearest-neighbor based approach called the KSG estimator and is designed for
three major tasks: (i) Classification Performance. (ii) Feature
Interpretability. (iii) Classifier Generalization. MRwMR-BUR-CLF estimates UR
via a classifier based approach. It adapts UR to different classifiers, further
improving the competitiveness of MRwMR-BUR for classification performance
oriented tasks. The performance of both MRwMR-BUR-KSG and MRwMR-BUR-CLF is
validated via experiments using six public datasets and three popular
classifiers. Specifically, as compared to MRwMR, the proposed MRwMR-BUR-KSG
improves the test accuracy by 2% - 3% with 25% - 30% fewer features being
selected, without increasing the algorithm complexity. MRwMR-BUR-CLF further
improves the classification performance by 3.8%- 5.5% (relative to MRwMR), and
it also outperforms three popular classifier dependent feature selection
methods.

本文提出了一种通过增加独特关联系统来改善高维数据集下特征选择的方法，并提出了两个方案来评估独特关联性。实验结果表明，提出的方法可以在减少特征数量的同时提高分类性能，其对分类器性能的依赖性相对较低。