Model heterogeneous federated learning (MHeteroFL) enables FL clients to
collaboratively train models with heterogeneous structures in a distributed
fashion. However, existing MHeteroFL methods rely on training loss to transfer
knowledge between the client model and the server model, resulting in limited
knowledge exchange. To address this limitation, we propose the Federated model
heterogeneous Matryoshka Representation Learning (FedMRL) approach for
supervised learning tasks. It adds an auxiliary small homogeneous model shared
by clients with heterogeneous local models. (1) The generalized and
personalized representations extracted by the two models' feature extractors
are fused by a personalized lightweight representation projector. This step
enables representation fusion to adapt to local data distribution. (2) The
fused representation is then used to construct Matryoshka representations with
multi-dimensional and multi-granular embedded representations learned by the
global homogeneous model header and the local heterogeneous model header. This
step facilitates multi-perspective representation learning and improves model
learning capability. Theoretical analysis shows that FedMRL achieves a $O(1/T)$
non-convex convergence rate. Extensive experiments on benchmark datasets
demonstrate its superior model accuracy with low communication and
computational costs compared to seven state-of-the-art baselines. It achieves
up to 8.48% and 24.94% accuracy improvement compared with the state-of-the-art
and the best same-category baseline, respectively.

Federated model heterogeneous Matryoshka Representation Learning (FedMRL) enables knowledge exchange and improves model learning capability in heterogeneous federated learning through representation fusion.