Various forms of regularization in learning tasks strive for different
notions of simplicity. This paper presents a spectral regularization technique,
which attaches a unique inductive bias to sequence modeling based on an
intuitive concept of simplicity defined in the Chomsky hierarchy. From
fundamental connections between Hankel matrices and regular grammars, we
propose to use the trace norm of the Hankel matrix, the tightest convex
relaxation of its rank, as the spectral regularizer. To cope with the fact that
the Hankel matrix is bi-infinite, we propose an unbiased stochastic estimator
for its trace norm. Ultimately, we demonstrate experimental results on Tomita
grammars, which exhibit the potential benefits of spectral regularization and
validate the proposed stochastic estimator.

本文介绍了一种基于 Chomsky 层次的直观简洁性概念，利用 Hankel 矩阵的跟踪范数作为频谱正则化器的高级模型，并提出了一种偏袒随机估计器以应对 Hankel 矩阵是双无限的事实，最终证明了频谱正则化在 Tomita 语法上的潜在优势以及提议的随机估计器的有效性。

光谱正则化：序列建模的归纳偏置

Spectral Regularization: an Inductive Bias for Sequence Modeling

Reliable generalization lies at the heart of safe ML and AI. However,
understanding when and how neural networks generalize remains one of the most
important unsolved problems in the field. In this work, we conduct an extensive
empirical study (20'910 models, 15 tasks) to investigate whether insights from
the theory of computation can predict the limits of neural network
generalization in practice. We demonstrate that grouping tasks according to the
Chomsky hierarchy allows us to forecast whether certain architectures will be
able to generalize to out-of-distribution inputs. This includes negative
results where even extensive amounts of data and training time never lead to
any non-trivial generalization, despite models having sufficient capacity to
fit the training data perfectly. Our results show that, for our subset of
tasks, RNNs and Transformers fail to generalize on non-regular tasks, LSTMs can
solve regular and counter-language tasks, and only networks augmented with
structured memory (such as a stack or memory tape) can successfully generalize
on context-free and context-sensitive tasks.

通过在 20,910 个模型和 15 个任务上进行广泛的实证研究，我们展示了按照 Chomsky 层次分类任务可以预测特定体系结构的广义能力。结果表明，只有增加了结构化记忆的网络（例如堆栈或内存带）才能成功地推广上下文自由和上下文相关的任务。