Neural networks offer good approximation to many tasks but consistently fail
to reach perfect generalization, even when theoretical work shows that such
perfect solutions can be expressed by certain architectures. Using the task of
formal language learning, we focus on one simple formal language and show that
the theoretically correct solution is in fact not an optimum of commonly used
objectives -- even with regularization techniques that according to common
wisdom should lead to simple weights and good generalization (L1, L2) or other
meta-heuristics (early-stopping, dropout). However, replacing standard targets
with the Minimum Description Length objective (MDL) results in the correct
solution being an optimum.

神经网络在形式语言学习任务中表现出了较好的逼近能力，然而，理论上证明了某些结构可以达到完美的广义，但实际上常用的目标函数（如 L1、L2 正则化、early-stopping 和 dropout）并不能得到理论上正确的解，而采用最小描述长度目标函数则能达到最优解。