This paper introduces novel alternate training procedures for hard-parameter
sharing Multi-Task Neural Networks (MTNNs). Traditional MTNN training faces
challenges in managing conflicting loss gradients, often yielding sub-optimal
performance. The proposed alternate training method updates shared and
task-specific weights alternately, exploiting the multi-head architecture of
the model. This approach reduces computational costs, enhances training
regularization, and improves generalization. Convergence properties similar to
those of the classical stochastic gradient method are established. Empirical
experiments demonstrate delayed overfitting, improved prediction, and reduced
computational demands. In summary, our alternate training procedures offer a
promising advancement for the training of hard-parameter sharing MTNNs.

该论文介绍了一种用于训练难参数共享的多任务神经网络的新型交替训练程序，该方法通过交替更新共享权重和任务特定权重，利用模型的多头架构，从而降低计算成本，增强训练规范化，并提高泛化性能。实证实验表明，该交替训练程序延迟过拟合，提高了预测准确性，降低了计算需求，对难参数共享的多任务神经网络的训练具有重要意义。

多任务神经网络共享与任务特定参数的轮替训练

Alternate Training of Shared and Task-Specific Parameters for Multi-Task  Neural Networks

It can be challenging to train multi-task neural networks that outperform or
even match their single-task counterparts. To help address this, we propose
using knowledge distillation where single-task models teach a multi-task model.
We enhance this training with teacher annealing, a novel method that gradually
transitions the model from distillation to supervised learning, helping the
multi-task model surpass its single-task teachers. We evaluate our approach by
multi-task fine-tuning BERT on the GLUE benchmark. Our method consistently
improves over standard single-task and multi-task training.

使用知识蒸馏和教师退火的方法，可帮助多任务神经网络训练并超越单任务神经网络，文中使用该方法在 GLUE 基准测试上成功的提升了 BERT 的多任务微调性能。