Multi-objective optimization (MOO) is receiving more attention in various
fields such as multi-task learning. Recent works provide some effective
algorithms with theoretical analysis but they are limited by the standard
$L$-smooth or bounded-gradient assumptions, which are typically unsatisfactory
for neural networks, such as recurrent neural networks (RNNs) and transformers.
In this paper, we study a more general and realistic class of $\ell$-smooth
loss functions, where $\ell$ is a general non-decreasing function of gradient
norm. We develop two novel single-loop algorithms for $\ell$-smooth MOO
problems, Generalized Smooth Multi-objective Gradient descent (GSMGrad) and its
stochastic variant, Stochastic Generalized Smooth Multi-objective Gradient
descent (SGSMGrad), which approximate the conflict-avoidant (CA) direction that
maximizes the minimum improvement among objectives. We provide a comprehensive
convergence analysis of both algorithms and show that they converge to an
$\epsilon$-accurate Pareto stationary point with a guaranteed $\epsilon$-level
average CA distance (i.e., the gap between the updating direction and the CA
direction) over all iterations, where totally $\mathcal{O}(\epsilon^{-2})$ and
$\mathcal{O}(\epsilon^{-4})$ samples are needed for deterministic and
stochastic settings, respectively. Our algorithms can also guarantee a tighter
$\epsilon$-level CA distance in each iteration using more samples. Moreover, we
propose a practical variant of GSMGrad named GSMGrad-FA using only
constant-level time and space, while achieving the same performance guarantee
as GSMGrad. Our experiments validate our theory and demonstrate the
effectiveness of the proposed methods.

本文研究了多目标优化问题中更一般且更现实的平滑损失函数类别，在神经网络中，提出了两种新颖的单循环算法 GSMGrad 和 SGSMGrad，以逼近在所有目标之间最大化最小改进的冲突避免方向。