Despite the capacity of neural nets to learn arbitrary functions, models
trained through gradient descent often exhibit a bias towards ``simpler''
functions. Various notions of simplicity have been introduced to characterize
this behavior. Here, we focus on the case of neural networks with discrete
(zero-one) inputs through the lens of their Fourier (Walsh-Hadamard)
transforms, where the notion of simplicity can be captured through the
\emph{degree} of the Fourier coefficients. We empirically show that neural
networks have a tendency to learn lower-degree frequencies. We show how this
spectral bias towards simpler features can in fact \emph{hurt} the neural
network's generalization on real-world datasets. To remedy this we propose a
new scalable functional regularization scheme that aids the neural network to
learn higher degree frequencies. Our regularizer also helps avoid erroneous
identification of low-degree frequencies, which further improves
generalization. We extensively evaluate our regularizer on synthetic datasets
to gain insights into its behavior. Finally, we show significantly improved
generalization on four different datasets compared to standard neural networks
and other relevant baselines.

研究神经网络中的频谱偏差问题，提出一种新的可扩展的函数正则化方案，以帮助神经网络学习更高阶的频率，并对四个不同的数据集进行了广泛的评估，显示相对于标准神经网络和其他基线方法显著提高了推广性。

一种可扩展的 Walsh-Hadamard 正则化器，以克服神经网络的低阶谱偏倚

A Scalable Walsh-Hadamard Regularizer to Overcome the Low-degree  Spectral Bias of Neural Networks

Recent work has shown that the input-output behavior of some machine learning
systems can be captured symbolically using Boolean expressions or tractable
Boolean circuits, which facilitates reasoning about the behavior of these
systems. While most of the focus has been on systems with Boolean inputs and
outputs, we address systems with discrete inputs and outputs, including ones
with discretized continuous variables as in systems based on decision trees. We
also focus on the suitability of encodings for computing prime implicants,
which have recently played a central role in explaining the decisions of
machine learning systems. We show some key distinctions with encodings for
satisfiability, and propose an encoding that is sound and complete for the
given task.

本文研究了带有离散输入和输出的机器学习系统在布尔表达式或可处理的布尔电路中的行为，并提出了一种可计算主析取项的编码方法，该方法是针对这种任务的一个完整有效的方案。