$L_0$ regularization of neural networks is a fundamental problem. In addition
to regularizing models for better generalizability, $L_0$ regularization also
applies to selecting input features and training sparse neural networks. There
is a large body of research on related topics, some with quite complicated
methods. In this paper, we show that a straightforward formulation, BinMask,
which multiplies weights with deterministic binary masks and uses the identity
straight-through estimator for backpropagation, is an effective $L_0$
regularizer. We evaluate BinMask on three tasks: feature selection, network
sparsification, and model regularization. Despite its simplicity, BinMask
achieves competitive performance on all the benchmarks without task-specific
tuning compared to methods designed for each task. Our results suggest that
decoupling weights from mask optimization, which has been widely adopted by
previous work, is a key component for effective $L_0$ regularization.

本文提出了一种名为 BinMask 的简单 $L_0$ 正则化方法，该方法采用确定性二进制掩码相乘的权重并使用恒等误差反向传播估计器，在特征选择、网络稀疏化和模型正则化等任务上均取得了竞争性的性能表现。