Multi-task learning (MTL) models have demonstrated impressive results in
computer vision, natural language processing, and recommender systems. Even
though many approaches have been proposed, how well these approaches balance
different tasks on each parameter still remains unclear. In this paper, we
propose to measure the task dominance degree of a parameter by the total
updates of each task on this parameter. Specifically, we compute the total
updates by the exponentially decaying Average of the squared Updates (AU) on a
parameter from the corresponding task.Based on this novel metric, we observe
that many parameters in existing MTL methods, especially those in the higher
shared layers, are still dominated by one or several tasks. The dominance of AU
is mainly due to the dominance of accumulative gradients from one or several
tasks. Motivated by this, we propose a Task-wise Adaptive learning rate
approach, AdaTask in short, to separate the \emph{accumulative gradients} and
hence the learning rate of each task for each parameter in adaptive learning
rate approaches (e.g., AdaGrad, RMSProp, and Adam). Comprehensive experiments
on computer vision and recommender system MTL datasets demonstrate that AdaTask
significantly improves the performance of dominated tasks, resulting SOTA
average task-wise performance. Analysis on both synthetic and real-world
datasets shows AdaTask balance parameters in every shared layer well.

本文提出了一种通过计算任务在某个参数上的累积更新量来度量参数在各个任务之间的占用程度的度量标准，并以此为基础提出了一种名为 AdaTask 的任务自适应学习率方法，该方法可以有效平衡多任务学习中共享层中的参数，对计算机视觉和推荐系统 MTL 数据集进行的全面实验表明，AdaTask 显著提高了受到支配的任务的性能，实现了 SOTA 平均任务性能。

AdaTask：一种面向任务的自适应学习率多任务学习方法

AdaTask: A Task-aware Adaptive Learning Rate Approach to Multi-task  Learning

Existing deep multitask learning (MTL) approaches align layers shared between
tasks in a parallel ordering. Such an organization significantly constricts the
types of shared structure that can be learned. The necessity of parallel
ordering for deep MTL is first tested by comparing it with permuted ordering of
shared layers. The results indicate that a flexible ordering can enable more
effective sharing, thus motivating the development of a soft ordering approach,
which learns how shared layers are applied in different ways for different
tasks. Deep MTL with soft ordering outperforms parallel ordering methods across
a series of domains. These results suggest that the power of deep MTL comes
from learning highly general building blocks that can be assembled to meet the
demands of each task.

测试表明，柔性排序能够更有效地实现共享结构，促进了软排序方法的发展。软排序的深度多任务学习在多个领域中表现优于并行排序方法，这表明深度多任务学习的力量来自于学习高度通用的构件，可以组装以满足每个任务的需求。