Model heterogeneous federated learning (MHeteroFL) enables FL clients to
collaboratively train models with heterogeneous structures in a distributed
fashion. However, existing MHeteroFL methods rely on training loss to transfer
knowledge between the client model and the server model, resulting in limited
knowledge exchange. To address this limitation, we propose the Federated model
heterogeneous Matryoshka Representation Learning (FedMRL) approach for
supervised learning tasks. It adds an auxiliary small homogeneous model shared
by clients with heterogeneous local models. (1) The generalized and
personalized representations extracted by the two models' feature extractors
are fused by a personalized lightweight representation projector. This step
enables representation fusion to adapt to local data distribution. (2) The
fused representation is then used to construct Matryoshka representations with
multi-dimensional and multi-granular embedded representations learned by the
global homogeneous model header and the local heterogeneous model header. This
step facilitates multi-perspective representation learning and improves model
learning capability. Theoretical analysis shows that FedMRL achieves a $O(1/T)$
non-convex convergence rate. Extensive experiments on benchmark datasets
demonstrate its superior model accuracy with low communication and
computational costs compared to seven state-of-the-art baselines. It achieves
up to 8.48% and 24.94% accuracy improvement compared with the state-of-the-art
and the best same-category baseline, respectively.

Federated model heterogeneous Matryoshka Representation Learning (FedMRL) enables knowledge exchange and improves model learning capability in heterogeneous federated learning through representation fusion.

联合模型异构母娃表示学习

Federated Model Heterogeneous Matryoshka Representation Learning

Common approaches rely on fixed-length embedding vectors from language models
as sentence embeddings for downstream tasks such as semantic textual similarity
(STS). Such methods are limited in their flexibility due to unknown
computational constraints and budgets across various applications. Matryoshka
Representation Learning (MRL) (Kusupati et al., 2022) encodes information at
finer granularities, i.e., with lower embedding dimensions, to adaptively
accommodate ad hoc tasks. Similar accuracy can be achieved with a smaller
embedding size, leading to speedups in downstream tasks. Despite its improved
efficiency, MRL still requires traversing all Transformer layers before
obtaining the embedding, which remains the dominant factor in time and memory
consumption. This prompts consideration of whether the fixed number of
Transformer layers affects representation quality and whether using
intermediate layers for sentence representation is feasible. In this paper, we
introduce a novel sentence embedding model called Two-dimensional Matryoshka
Sentence Embedding (2DMSE). It supports elastic settings for both embedding
sizes and Transformer layers, offering greater flexibility and efficiency than
MRL. We conduct extensive experiments on STS tasks and downstream applications.
The experimental results demonstrate the effectiveness of our proposed model in
dynamically supporting different embedding sizes and Transformer layers,
allowing it to be highly adaptable to various scenarios.

引入了一种名为二维母婴句子嵌入的新型句子嵌入模型，支持弹性设置嵌入尺寸和 Transformer 层数，具备比 Matryoshka Representation Learning 更大的灵活性和效率。在语义文本相似性（STS）任务和下游应用上进行了涉及实验，实验结果表明我们提出的模型在动态支持不同嵌入尺寸和 Transformer 层数方面表现出了显著的效果，能够高度适应各种场景。