Tasks in multi-task learning often correlate, conflict, or even compete with each other. As a result, a single solution that is optimal for all tasks rarely exists. Recent papers introduced the concept of Pareto optimality to this field and directly cast multi-task learning as multi-objective optimization problems, but solutions returned by existing methods are typically finite, sparse, and discrete. We present a novel, efficient method that generates locally continuous Pareto sets and Pareto fronts, which opens up the possibility of continuous analysis of Pareto optimal solutions in machine learning problems. We scale up theoretical results in multi-objective optimization to modern machine learning problems by proposing a sample-based sparse linear system, for which standard Hessian-free solvers in machine learning can be applied. We compare our method to the state-of-the-art algorithms and demonstrate its usage of analyzing local Pareto sets on various multi-task classification and regression problems. The experimental results confirm that our algorithm reveals the primary directions in local Pareto sets for trade-off balancing, finds more solutions with different trade-offs efficiently, and scales well to tasks with millions of parameters.

提出了一种新的、高效的方法，可以生成局部连续的Pareto集和Pareto fronts，并将其应用于现代机器学习问题中。通过提出基于样本的稀疏线性系统，将多目标优化的理论结果扩展到现代机器学习问题中，并实现了局部Pareto集的分析。与现有算法相比，通过在各种多任务分类和回归问题上的应用，证明了我们的算法在平衡权衡、有效地找到更多的不同权衡解以及迎合百万级参数任务的能力。

多任务学习中高效的连续 Pareto 探索