Vertical Federated Learning (VFL) has gained increasing attention as a novel
training paradigm that integrates sample alignment and feature union. However,
existing VFL methods face challenges when dealing with heterogeneous local
models among participants, which affects optimization convergence and
generalization. To address this issue, this paper proposes a novel approach
called Vertical Federated learning for training Multi-parties Heterogeneous
models (VFedMH). VFedMH focuses on aggregating the embeddings of each
participant's knowledge instead of intermediate results during forward
propagation. The active party, who possesses labels and features of the sample,
in VFedMH securely aggregates local embeddings to obtain global knowledge
embeddings, and sends them to passive parties. The passive parties, who own
only features of the sample, then utilize the global embeddings to propagate
forward on their local heterogeneous networks. However, the passive party does
not own the labels, so the local model gradient cannot be calculated locally.
To overcome this limitation, the active party assists the passive party in
computing its local heterogeneous model gradients. Then, each participant
trains their local model using the heterogeneous model gradients. The objective
is to minimize the loss value of their respective local heterogeneous models.
Additionally, the paper provides a theoretical analysis of VFedMH's convergence
performance. Extensive experiments are conducted to demonstrate that VFedMH can
simultaneously train multiple heterogeneous models with heterogeneous
optimization and outperform some recent methods in model performance.

垂直联邦学习（VFL）作为一种集成样本对齐和特征汇聚的新型训练范例，受到了越来越多的关注。然而，现有的 VFL 方法在处理参与者之间的异构本地模型时面临挑战，影响了优化收敛和泛化能力。为了解决这个问题，本文提出了一种新颖的方法，称为垂直联邦学习用于训练多方异构模型（VFedMH）。VFedMH 专注于聚合每个参与者的知识嵌入，而不是在前向传播过程中聚合中间结果。通过安全地聚合本地嵌入获取全局知识嵌入，并将其发送给被动方。然后，被动方利用全局嵌入在本地异构网络上进行前向传播。然而，被动方没有标签，因此无法在本地计算本地模型的梯度。为了克服这个局限性，主动方协助被动方计算其本地异构模型梯度。然后，每个参与者使用异构模型梯度训练其本地模型，目标是最小化各自本地异构模型的损失值。此外，本文还对 VFedMH 的收敛性能进行了理论分析。通过大量实验证明，VFedMH 能够同时训练多个异构模型，并在模型性能方面超过一些最近的方法。