Using backward error analysis, we compute implicit training biases in
multitask and continual learning settings for neural networks trained with
stochastic gradient descent. In particular, we derive modified losses that are
implicitly minimized during training. They have three terms: the original loss,
accounting for convergence, an implicit flatness regularization term
proportional to the learning rate, and a last term, the conflict term, which
can theoretically be detrimental to both convergence and implicit
regularization. In multitask, the conflict term is a well-known quantity,
measuring the gradient alignment between the tasks, while in continual learning
the conflict term is a new quantity in deep learning optimization, although a
basic tool in differential geometry: The Lie bracket between the task
gradients.

使用反向误差分析为神经网络在多任务学习和持续学习设置中计算隐含的训练偏差，并推导出通过随机梯度下降训练时隐含地最小化的修改后的损失函数，其中包括原始损失、隐含的平直度正则化项与冲突项。这里，冲突项可理论上对收敛和隐含正则化产生不良影响，在多任务学习中，冲突项是一个已知量，用于测量任务之间的梯度对齐性，而在持续学习中，冲突项是深度学习优化中的一个新量，尽管在微分几何中是一个基本工具：任务梯度之间的李括号。