LASSO regularization is a popular regression tool to enhance the prediction
accuracy of statistical models by performing variable selection through the
$\ell_1$ penalty, initially formulated for the linear model and its variants.
In this paper, the territory of LASSO is extended to two-layer ReLU neural
networks, a fashionable and powerful nonlinear regression model. Specifically,
given a neural network whose output $y$ depends only on a small subset of input
$\boldsymbol{x}$, denoted by $\mathcal{S}^{\star}$, we prove that the LASSO
estimator can stably reconstruct the neural network and identify
$\mathcal{S}^{\star}$ when the number of samples scales logarithmically with
the input dimension. This challenging regime has been well understood for
linear models while barely studied for neural networks. Our theory lies in an
extended Restricted Isometry Property (RIP)-based analysis framework for
two-layer ReLU neural networks, which may be of independent interest to other
LASSO or neural network settings. Based on the result, we advocate a neural
network-based variable selection method. Experiments on simulated and
real-world datasets show promising performance of the variable selection
approach compared with existing techniques.

本文将 LASSO 正则化技术推广到两层 ReLU 神经网络中，通过扩展限制等距性原理分析框架，可实现稳定重构神经网络并识别输入的子集，从而提出了一种基于神经网络的变量选取方法。实验结果表明与现有技术相比，该方法具有良好表现。

通过 LASSO 正则化证明双层 ReLU 神经网络的可识别性

Provable Identifiability of Two-Layer ReLU Neural Networks via LASSO  Regularization

We investigate methods for combining multiple self-supervised tasks--i.e.,
supervised tasks where data can be collected without manual labeling--in order
to train a single visual representation. First, we provide an apples-to-apples
comparison of four different self-supervised tasks using the very deep
ResNet-101 architecture. We then combine tasks to jointly train a network. We
also explore lasso regularization to encourage the network to factorize the
information in its representation, and methods for "harmonizing" network inputs
in order to learn a more unified representation. We evaluate all methods on
ImageNet classification, PASCAL VOC detection, and NYU depth prediction. Our
results show that deeper networks work better, and that combining tasks--even
via a naive multi-head architecture--always improves performance. Our best
joint network nearly matches the PASCAL performance of a model pre-trained on
ImageNet classification, and matches the ImageNet network on NYU depth
prediction.

研究多种自监督学习任务相结合的方法，用于训练单一的视觉表征，并在 ImageNet 分类、PASCAL VOC 检测和 NYU 深度预测上获得了优秀的结果。

多任务自监督视觉学习

Multi-task Self-Supervised Visual Learning

The goal of this paper is to contrast and survey the major advances in two of
the most commonly used high-dimensional techniques, namely, the Lasso and
horseshoe regularization. Lasso is a gold standard for predictor selection
while horseshoe is a state-of-the-art Bayesian estimator for sparse signals.
Lasso is fast and scalable and uses convex optimization whilst the horseshoe is
non-convex. Our novel perspective focuses on three aspects: (i) theoretical
optimality in high dimensional inference for the Gaussian sparse model and
beyond, (ii) efficiency and scalability of computation and (iii) methodological
development and performance.

本文对两种常用的高维技术进行对比和调查，即 Lasso 与马蹄铁正则化技术，旨在理论上优化高维推理，提高计算效率和可扩展性，以及方法论的发展和性能方面。