Training a single model on multiple input domains and/or output tasks allows
for compressing information from multiple sources into a unified backbone hence
improves model efficiency. It also enables potential positive knowledge
transfer across tasks/domains, leading to improved accuracy and data-efficient
training. However, optimizing such networks is a challenge, in particular due
to discrepancies between the different tasks or domains: Despite several
hypotheses and solutions proposed over the years, recent work has shown that
uniform scalarization training, i.e., simply minimizing the average of the task
losses, yields on-par performance with more costly SotA optimization methods.
This raises the issue of how well we understand the training dynamics of
multi-task and multi-domain networks. In this work, we first devise a
large-scale unified analysis of multi-domain and multi-task learning to better
understand the dynamics of scalarization across varied task/domain combinations
and model sizes. Following these insights, we then propose to leverage
population-based training to efficiently search for the optimal scalarization
weights when dealing with a large number of tasks or domains.

通过多领域和多任务学习的大规模统一分析，深入了解标量化在各种任务和领域组合以及模型规模中的动态，然后借助基于群体的训练来有效地搜索处理大量任务或领域时的最优标量化权重。