Ridgeless regression has garnered attention among researchers, particularly
in light of the ``Benign Overfitting'' phenomenon, where models interpolating
noisy samples demonstrate robust generalization. However, kernel ridgeless
regression does not always perform well due to the lack of flexibility. This
paper enhances kernel ridgeless regression with Locally-Adaptive-Bandwidths
(LAB) RBF kernels, incorporating kernel learning techniques to improve
performance in both experiments and theory. For the first time, we demonstrate
that functions learned from LAB RBF kernels belong to an integral space of
Reproducible Kernel Hilbert Spaces (RKHSs). Despite the absence of explicit
regularization in the proposed model, its optimization is equivalent to solving
an $\ell_0$-regularized problem in the integral space of RKHSs, elucidating the
origin of its generalization ability. Taking an approximation analysis
viewpoint, we introduce an $l_q$-norm analysis technique (with $0<q<1$) to
derive the learning rate for the proposed model under mild conditions. This
result deepens our theoretical understanding, explaining that our algorithm's
robust approximation ability arises from the large capacity of the integral
space of RKHSs, while its generalization ability is ensured by sparsity,
controlled by the number of support vectors. Experimental results on both
synthetic and real datasets validate our theoretical conclusions.

本文通过加入局部自适应带宽的 RBF 核和核学习技术，提升了核无岭回归，并首次证明了由 LAB RBF 核学习的函数属于可重构核希尔伯特空间。尽管所提出模型中没有显式正则化，但其优化等效于在可重构核希尔伯特空间中解决一个 l0 正则化问题，解释了其泛化能力的来源。在近似分析角度下，我们引入了一种 lq - 范数分析技术（其中 0<q<1），以在适度条件下推导所提模型的学习率。实验结果在合成和真实数据集上验证了我们的理论推论。

非对称核学习的核岭回归学习分析

Learning Analysis of Kernel Ridgeless Regression with Asymmetric Kernel  Learning

The lack of sufficient flexibility is the key bottleneck of kernel-based
learning that relies on manually designed, pre-given, and non-trainable
kernels. To enhance kernel flexibility, this paper introduces the concept of
Locally-Adaptive-Bandwidths (LAB) as trainable parameters to enhance the Radial
Basis Function (RBF) kernel, giving rise to the LAB RBF kernel. The parameters
in LAB RBF kernels are data-dependent, and its number can increase with the
dataset, allowing for better adaptation to diverse data patterns and enhancing
the flexibility of the learned function. This newfound flexibility also brings
challenges, particularly with regards to asymmetry and the need for an
efficient learning algorithm. To address these challenges, this paper for the
first time establishes an asymmetric kernel ridge regression framework and
introduces an iterative kernel learning algorithm. This novel approach not only
reduces the demand for extensive support data but also significantly improves
generalization by training bandwidths on the available training data.
Experimental results on real datasets underscore the remarkable performance of
the proposed algorithm, showcasing its superior capability in handling
large-scale datasets compared to Nystr\"om approximation-based algorithms.
Moreover, it demonstrates a significant improvement in regression accuracy over
existing kernel-based learning methods and even surpasses residual neural
networks.

本文介绍了一种新型的局部自适应带宽（LAB）径向基函数（RBF）核，利用可训练参数增强了核的灵活性，同时建立了一个非对称核岭回归框架并引入了一种迭代式核学习算法，实验结果表明该算法在处理大规模数据集上具有显著性能优势，并且在回归准确度上优于现有的基于核的学习方法，甚至超过了残差神经网络。