Neural networks exhibit simplicity bias; they rely on simpler features while
ignoring equally predictive but more complex features. In this work, we
introduce a novel approach termed imbalanced label coupling to investigate
scenarios where simple and complex features exhibit different levels of
predictive power. In these cases, complex features still contribute to
predictions. The trained networks make predictions in alignment with the
ascending complexity of input features according to how they correlate with the
label in the training set, irrespective of the underlying predictive power. For
instance, even when simple spurious features distort predictions in CIFAR-10,
most cats are predicted to be dogs, and most trucks are predicted to be
automobiles! This observation provides direct evidence that the neural network
learns core features in the presence of spurious features. We empirically show
that last-layer retraining with target data distribution is effective, yet
insufficient to fully recover core features when spurious features are
perfectly correlated with the target labels in our synthetic dataset. We hope
our research contributes to a deeper understanding of the implicit bias of
neural networks.

神经网络在存在偏执特征时，能够学习到核心特征，即使简单虚假特征扭曲了预测结果；研究通过不平衡标签耦合的方法，调查了简单和复杂特征在预测中的不同水平，以及复杂特征在预测中的贡献。

神经网络是隐式决策树：层次简约偏差

Neural Networks Are Implicit Decision Trees: The Hierarchical Simplicity  Bias

Recent years have seen a shift from a pattern mining process that has users
define constraints before-hand, and sift through the results afterwards, to an
interactive one. This new framework depends on exploiting user feedback to
learn a quality function for patterns. Existing approaches have a weakness in
that they use static pre-defined low-level features, and attempt to learn
independent weights representing their importance to the user. As an
alternative, we propose to work with more complex features that are derived
directly from the pattern ranking imposed by the user. Learned weights are then
aggregated onto lower-level features and help to drive the quality function in
the right direction. We explore the effect of different parameter choices
experimentally and find that using higher-complexity features leads to the
selection of patterns that are better aligned with a hidden quality function
while not adding significantly to the run times of the method.
Getting good user feedback requires to quickly present diverse patterns,
something that we achieve but pushing an existing diversity constraint into the
sampling component of the interactive mining system LetSip. Resulting patterns
allow in most cases to converge to a good solution more quickly.
Combining the two improvements, finally, leads to an algorithm showing clear
advantages over the existing state-of-the-art.

本文讨论了如何通过用户反馈学习模式的质量函数，并提出了使用更复杂的直接从模式排名派生的特征来替代现有方法中的低级特征，以及如何将多样性约束纳入交互式挖掘系统中。实验结果表明，在参数选择和特征工程方面的优化会提高挖掘算法的性能。

利用复杂的模式特征实现交互式模式挖掘

Exploiting complex pattern features for interactive pattern mining

This paper considers the power of deep neural networks (deep nets for short)
in realizing data features. Based on refined covering number estimates, we find
that, to realize some complex data features, deep nets can improve the
performances of shallow neural networks (shallow nets for short) without
requiring additional capacity costs. This verifies the advantage of deep nets
in realizing complex features. On the other hand, to realize some simple data
feature like the smoothness, we prove that, up to a logarithmic factor, the
approximation rate of deep nets is asymptotically identical to that of shallow
nets, provided that the depth is fixed. This exhibits a limitation of deep nets
in realizing simple features.

本文研究深度神经网络在实现数据特征方面的优势，发现相比浅层神经网络，深度神经网络能够在不需要额外容量开销的情况下，提高实现某些复杂特征的性能，但在实现一些简单特征方面，深度神经网络的逼近率与浅层神经网络的相同，呈对数级别，在固定深度的情况下，存在一定的局限性。