Optimizing the performance of many objectives (instantiated by tasks or
clients) jointly with a few Pareto stationary solutions (models) is critical in
machine learning. However, previous multi-objective optimization methods often
focus on a few number of objectives and cannot scale to many objectives that
outnumber the solutions, leading to either subpar performance or ignored
objectives. We introduce Many-objective multi-solution Transport (MosT), a
framework that finds multiple diverse solutions in the Pareto front of many
objectives. Our insight is to seek multiple solutions, each performing as a
domain expert and focusing on a specific subset of objectives while
collectively covering all of them. MosT formulates the problem as a bi-level
optimization of weighted objectives for each solution, where the weights are
defined by an optimal transport between the objectives and solutions. Our
algorithm ensures convergence to Pareto stationary solutions for complementary
subsets of objectives. On a range of applications in federated learning,
multi-task learning, and mixture-of-prompt learning for LLMs, MosT distinctly
outperforms strong baselines, delivering high-quality, diverse solutions that
profile the entire Pareto frontier, thus ensuring balanced trade-offs across
many objectives.

通过寻求多个多目标优化问题的互补子集解决方案的权衡，我们引入了一种称为 MosT 的多目标多解传输框架，其通过在每个解决方案中对加权目标进行双层优化，并使用目标和解决方案之间的最优传输来定义权重，确保收敛到 Pareto 稳态解决方案。在联邦学习、多任务学习和混合提示学习等应用中，MosT 都显著优于强基线方法，提供高质量的多样化解决方案，刻画了整个 Pareto 前沿，从而实现了在多个目标之间的平衡权衡。