The ability to generalize under distributional shifts is essential to
reliable machine learning, while models optimized with empirical risk
minimization usually fail on non-$i.i.d$ testing data. Recently, invariant
learning methods for out-of-distribution (OOD) generalization propose to find
causally invariant relationships with multi-environments. However, modern
datasets are frequently multi-sourced without explicit source labels, rendering
many invariant learning methods inapplicable. In this paper, we propose
Kernelized Heterogeneous Risk Minimization (KerHRM) algorithm, which achieves
both the latent heterogeneity exploration and invariant learning in kernel
space, and then gives feedback to the original neural network by appointing
invariant gradient direction. We theoretically justify our algorithm and
empirically validate the effectiveness of our algorithm with extensive
experiments.

本文介绍了一种基于核化的异构风险最小化算法，实现了潜在特性探索和不变学习，并通过指定不变梯度方向向原始神经网络传递反馈。我们从理论和实践两方面证明了我们的算法的有效性。

核化异构风险最小化

Kernelized Heterogeneous Risk Minimization

We investigate the connections between neural networks and simple building
blocks in kernel space. In particular, using well established feature space
tools such as direct sum, averaging, and moment lifting, we present an algebra
for creating "compositional" kernels from bags of features. We show that these
operations correspond to many of the building blocks of "neural tangent kernels
(NTK)". Experimentally, we show that there is a correlation in test error
between neural network architectures and the associated kernels. We construct a
simple neural network architecture using only 3x3 convolutions, 2x2 average
pooling, ReLU, and optimized with SGD and MSE loss that achieves 96% accuracy
on CIFAR10, and whose corresponding compositional kernel achieves 90% accuracy.
We also use our constructions to investigate the relative performance of neural
networks, NTKs, and compositional kernels in the small dataset regime. In
particular, we find that compositional kernels outperform NTKs and neural
networks outperform both kernel methods.

通过研究神经网络和内核空间中的简单构建块之间的联系，我们提出了一种从特征组中创建 “组合” 内核的代数。在实验中，我们发现神经网络体系结构和相关内核的测试误差之间存在相关性，并构建了一个只使用 3x3 卷积、2x2 平均池化、ReLU 并使用 SGD 和 MSE 损失进行优化的简单神经网络体系结构，在 CIFAR10 数据集上获得 96％的准确度，其对应的组合内核可达到 90％的准确度。我们还使用我们的构造方法研究了神经网络、NTK 和组合内核在小数据集环境中的相对性能表现，特别是发现组合内核优于 NTK，神经网络优于这两种内核方法。

无切线的神经核函数

Neural Kernels Without Tangents

Nonlinear similarity measures defined in kernel space, such as correntropy,
can extract higher-order statistics of data and offer potentially significant
performance improvement over their linear counterparts especially in
non-Gaussian signal processing and machine learning. In this work, we propose a
new similarity measure in kernel space, called the kernel risk-sensitive loss
(KRSL), and provide some important properties. We apply the KRSL to adaptive
filtering and investigate the robustness, and then develop the MKRSL algorithm
and analyze the mean square convergence performance. Compared with correntropy,
the KRSL can offer a more efficient performance surface, thereby enabling a
gradient based method to achieve faster convergence speed and higher accuracy
while still maintaining the robustness to outliers. Theoretical analysis
results and superior performance of the new algorithm are confirmed by
simulation.

本文提出了一种新的核空间相似度衡量方法 —— 核风险敏感损失，并将其应用于自适应滤波和鲁棒性研究，发现相比于 correntropy，KRSL 不仅能够在梯度法中提高收敛速度和精度，而且还能减少异常值的影响。理论分析结果和仿真实验证明了该算法的优异性能。