New models of the attention-based random forests called LARF (Leaf
Attention-based Random Forest) are proposed. The first idea behind the models
is to introduce a two-level attention, where one of the levels is the "leaf"
attention and the attention mechanism is applied to every leaf of trees. The
second level is the tree attention depending on the "leaf" attention. The
second idea is to replace the softmax operation in the attention with the
weighted sum of the softmax operations with different parameters. It is
implemented by applying a mixture of the Huber's contamination models and can
be regarded as an analog of the multi-head attention with "heads" defined by
selecting a value of the softmax parameter. Attention parameters are simply
trained by solving the quadratic optimization problem. To simplify the tuning
process of the models, it is proposed to make the tuning contamination
parameters to be training and to compute them by solving the quadratic
optimization problem. Many numerical experiments with real datasets are
performed for studying LARFs. The code of proposed algorithms can be found in
this https URL.

该论文提出了基于注意力的随机森林 LARF 模型，其将注意力机制分为两个级别 —— 叶子级别和树级别，并使用带有不同参数的 softmax 操作的加权和来代替注意力机制，通过解决二次优化问题来训练注意力参数和污染参数。