We present an adaptive step-size method, which does not include line-search
techniques, for solving a wide class of nonconvex multiobjective programming
problems on an unbounded constraint set. We also prove convergence of a general
approach under modest assumptions. More specifically, the convexity criterion
might not be satisfied by the objective function. Unlike descent line-search
algorithms, it does not require an initial step-size to be determined by a
previously determined Lipschitz constant. The process's primary characteristic
is its gradual step-size reduction up until a predetermined condition is met.
It can be specifically applied to offer an innovative multi-gradient projection
method for unbounded constrained optimization issues. Preliminary findings from
a few computational examples confirm the accuracy of the strategy. We apply the
proposed technique to some multi-task learning experiments to show its efficacy
for large-scale challenges.

我们提出了一种适应性步长方法，用于解决一个广泛类别的非凸多目标规划问题，且不包含线搜索技术，适用于无界约束集。我们证明了在适度假设下，该方法的收敛性，并应用该方法进行了一些实验以验证其有效性。

适应性多梯度方法在准凸向量优化和多任务学习中的应用

Adaptive multi-gradient methods for quasiconvex vector optimization and  applications to multi-task learning

The self-attention mechanism (SAM) is widely used in various fields of
artificial intelligence and has successfully boosted the performance of
different models. However, current explanations of this mechanism are mainly
based on intuitions and experiences, while there still lacks direct modeling
for how the SAM helps performance. To mitigate this issue, in this paper, based
on the dynamical system perspective of the residual neural network, we first
show that the intrinsic stiffness phenomenon (SP) in the high-precision
solution of ordinary differential equations (ODEs) also widely exists in
high-performance neural networks (NN). Thus the ability of NN to measure SP at
the feature level is necessary to obtain high performance and is an important
factor in the difficulty of training NN. Similar to the adaptive step-size
method which is effective in solving stiff ODEs, we show that the SAM is also a
stiffness-aware step size adaptor that can enhance the model's representational
ability to measure intrinsic SP by refining the estimation of stiffness
information and generating adaptive attention values, which provides a new
understanding about why and how the SAM can benefit the model performance. This
novel perspective can also explain the lottery ticket hypothesis in SAM, design
new quantitative metrics of representational ability, and inspire a new
theoretic-inspired approach, StepNet. Extensive experiments on several popular
benchmarks demonstrate that StepNet can extract fine-grained stiffness
information and measure SP accurately, leading to significant improvements in
various visual tasks.

基于动力系统视角和自适应步长方法，本文提出自注意力机制与内在分型现象以及刚性信息的测量与适应性关系，新的理论视角能够解释自注意机制对模型性能的影响，并且启发了一种新的算法 StepNet。大量实验证明 StepNet 能够提取细粒度的刚性信息并有效度量分型现象，从而在多种视觉任务中取得显著的性能改进。