Model heterogeneous federated learning (MHeteroFL) enables FL clients to
collaboratively train models with heterogeneous structures in a distributed
fashion. However, existing MHeteroFL methods rely on training loss to transfer
knowledge between the client model and the server model, resulting in limited
knowledge exchange. To address this limitation, we propose the Federated model
heterogeneous Matryoshka Representation Learning (FedMRL) approach for
supervised learning tasks. It adds an auxiliary small homogeneous model shared
by clients with heterogeneous local models. (1) The generalized and
personalized representations extracted by the two models' feature extractors
are fused by a personalized lightweight representation projector. This step
enables representation fusion to adapt to local data distribution. (2) The
fused representation is then used to construct Matryoshka representations with
multi-dimensional and multi-granular embedded representations learned by the
global homogeneous model header and the local heterogeneous model header. This
step facilitates multi-perspective representation learning and improves model
learning capability. Theoretical analysis shows that FedMRL achieves a $O(1/T)$
non-convex convergence rate. Extensive experiments on benchmark datasets
demonstrate its superior model accuracy with low communication and
computational costs compared to seven state-of-the-art baselines. It achieves
up to 8.48% and 24.94% accuracy improvement compared with the state-of-the-art
and the best same-category baseline, respectively.

Federated model heterogeneous Matryoshka Representation Learning (FedMRL) enables knowledge exchange and improves model learning capability in heterogeneous federated learning through representation fusion.

联合模型异构母娃表示学习

Federated Model Heterogeneous Matryoshka Representation Learning

While Transformer has become the de-facto standard for speech, modeling upon
the fine-grained frame-level features remains an open challenge of capturing
long-distance dependencies and distributing the attention weights. We propose
\textit{Progressive Down-Sampling} (PDS) which gradually compresses the
acoustic features into coarser-grained units containing more complete semantic
information, like text-level representation. In addition, we develop a
representation fusion method to alleviate information loss that occurs
inevitably during high compression. In this way, we compress the acoustic
features into 1/32 of the initial length while achieving better or comparable
performances on the speech recognition task. And as a bonus, it yields
inference speedups ranging from 1.20$\times$ to 1.47$\times$. By reducing the
modeling burden, we also achieve competitive results when training on the more
challenging speech translation task.

为了压缩声学特征并同时保留更丰富的信息，我们提出了逐步向下采样（PDS）的方法，并开发了一种表示融合方法来减少信息损失。可以将声学特征压缩到初始长度的 1/32，同时在语音识别任务上获得更好或相当的性能，并带来 1.20 倍至 1.47 倍的推理加速。我们还在更具挑战性的语音翻译任务训练时取得了竞争结果。

跨越声学建模的粒度鸿沟

Bridging the Granularity Gap for Acoustic Modeling

In order to quickly adapt to new data, few-shot learning aims at learning
from few examples, often by using already acquired knowledge. The new data
often differs from the previously seen data due to a domain shift, that is, a
change of the input-target distribution. While several methods perform well on
small domain shifts like new target classes with similar inputs, larger domain
shifts are still challenging. Large domain shifts may result in high-level
concepts that are not shared between the original and the new domain, whereas
low-level concepts like edges in images might still be shared and useful. For
cross-domain few-shot learning, we suggest representation fusion to unify
different abstraction levels of a deep neural network into one representation.
We propose Cross-domain Hebbian Ensemble Few-shot learning (CHEF), which
achieves representation fusion by an ensemble of Hebbian learners acting on
different layers of a deep neural network. Ablation studies show that
representation fusion is a decisive factor to boost cross-domain few-shot
learning. On the few-shot datasets miniImagenet and tieredImagenet with small
domain shifts, CHEF is competitive with state-of-the-art methods. On
cross-domain few-shot benchmark challenges with larger domain shifts, CHEF
establishes novel state-of-the-art results in all categories. We further apply
CHEF on a real-world cross-domain application in drug discovery. We consider a
domain shift from bioactive molecules to environmental chemicals and drugs with
twelve associated toxicity prediction tasks. On these tasks, that are highly
relevant for computational drug discovery, CHEF significantly outperforms all
its competitors. Github: this https URL

提出了一种称为 Cross-domain Hebbian Ensemble Few-shot learning (CHEF) 的方法，通过深度神经网络不同层次的 Hebbian 学习器进行集成表示融合，从而实现跨领域 Few-Shot Learning。在小域移位的 few-shot 数据集上，表现与现有方法有竞争力，在大域移位的 Few-Shot Learning 基准测试中，CHEF 建立了新的最佳结果，并在药物发现的跨领域应用中表现出更强的实用性。

表示融合的跨域少样本学习

Cross-Domain Few-Shot Learning by Representation Fusion

Recently, multi-view representation learning has become a rapidly growing
direction in machine learning and data mining areas. This paper introduces two
categories for multi-view representation learning: multi-view representation
alignment and multi-view representation fusion. Consequently, we first review
the representative methods and theories of multi-view representation learning
based on the perspective of alignment, such as correlation-based alignment.
Representative examples are canonical correlation analysis (CCA) and its
several extensions. Then from the perspective of representation fusion we
investigate the advancement of multi-view representation learning that ranges
from generative methods including multi-modal topic learning, multi-view sparse
coding, and multi-view latent space Markov networks, to neural network-based
methods including multi-modal autoencoders, multi-view convolutional neural
networks, and multi-modal recurrent neural networks. Further, we also
investigate several important applications of multi-view representation
learning. Overall, this survey aims to provide an insightful overview of
theoretical foundation and state-of-the-art developments in the field of
multi-view representation learning and to help researchers find the most
appropriate tools for particular applications.

本文介绍了多视图表示学习中的两个类别：多视图表示对齐和多视图表示融合，并从多个角度回顾了代表性的方法和理论，进而调查了多视图表示学习的发展。最终，探讨了多视图表示学习的几个重要应用，并旨在为研究人员提供深入的理论基础和最新的发展。