Multimodal federated learning (FL) aims to enrich model training in FL
settings where clients are collecting measurements across multiple modalities.
However, key challenges to multimodal FL remain unaddressed, particularly in
heterogeneous network settings where: (i) the set of modalities collected by
each client will be diverse, and (ii) communication limitations prevent clients
from uploading all their locally trained modality models to the server. In this
paper, we propose multimodal Federated learning with joint Modality and Client
selection (mmFedMC), a new FL methodology that can tackle the above-mentioned
challenges in multimodal settings. The joint selection algorithm incorporates
two main components: (a) A modality selection methodology for each client,
which weighs (i) the impact of the modality, gauged by Shapley value analysis,
(ii) the modality model size as a gauge of communication overhead, against
(iii) the frequency of modality model updates, denoted recency, to enhance
generalizability. (b) A client selection strategy for the server based on the
local loss of modality model at each client. Experiments on five real-world
datasets demonstrate the ability of mmFedMC to achieve comparable accuracy to
several baselines while reducing the communication overhead by over 20x. A demo
video of our methodology is available at this https URL

本文提出了一种新的多模型联邦学习方法（mmFedMC），它能够处理多模态设置中的挑战，实验结果表明该方法能够在减少通信开销的同时实现可比较的准确性。

高效沟通的多模态联邦学习：联合选择模态与客户端

Communication-Efficient Multimodal Federated Learning: Joint Modality  and Client Selection

Rapid advancements over the years have helped machine learning models reach
previously hard-to-achieve goals, sometimes even exceeding human capabilities.
However, to attain the desired accuracy, the model sizes and in turn their
computational requirements have increased drastically. Thus, serving
predictions from these models to meet any target latency and cost requirements
of applications remains a key challenge, despite recent work in building
inference-serving systems as well as algorithmic approaches that dynamically
adapt models based on inputs. In this paper, we introduce a form of dynamism,
modality selection, where we adaptively choose modalities from inference inputs
while maintaining the model quality. We introduce MOSEL, an automated inference
serving system for multi-modal ML models that carefully picks input modalities
per request based on user-defined performance and accuracy requirements. MOSEL
exploits modality configurations extensively, improving system throughput by
3.6$\times$ with an accuracy guarantee and shortening job completion times by
11$\times$.

本文介绍了一种自适应地选择多种模态的推理输入，以满足性能和精确性要求的自动化推理服务系统 MOSEL，通过广泛利用模态配置，提高系统吞吐量 3.6 倍，缩短作业完成时间 11 倍。

MOSEL：使用动态模态选择的推理服务

MOSEL: Inference Serving Using Dynamic Modality Selection

Modality selection is an important step when designing multimodal systems,
especially in the case of cross-domain activity recognition as certain
modalities are more robust to domain shift than others. However, selecting only
the modalities which have a positive contribution requires a systematic
approach. We tackle this problem by proposing an unsupervised modality
selection method (ModSelect), which does not require any ground-truth labels.
We determine the correlation between the predictions of multiple unimodal
classifiers and the domain discrepancy between their embeddings. Then, we
systematically compute modality selection thresholds, which select only
modalities with a high correlation and low domain discrepancy. We show in our
experiments that our method ModSelect chooses only modalities with positive
contributions and consistently improves the performance on a Synthetic-to-Real
domain adaptation benchmark, narrowing the domain gap.

本研究提出了一种无监督模态选择方法，即 ModSelect，通过计算多模态分类器的预测之间的相关性和其嵌入的域差异，从而系统地计算模态选择阈值，选择只具有高相关性和低域差异的模态，从而达到选择正向贡献模态的目的，并在合成到真实领域自适应基准测试中表现出稳健的性能提升。

ModSelect: 为合成与真实域通用化自动选择模态

ModSelect: Automatic Modality Selection for Synthetic-to-Real Domain  Generalization

Automatically generating a natural language sentence to describe the content
of an input video is a very challenging problem. It is an essential multimodal
task in which auditory and visual contents are equally important. Although
audio information has been exploited to improve video captioning in previous
works, it is usually regarded as an additional feature fed into a black box
fusion machine. How are the words in the generated sentences associated with
the auditory and visual modalities? The problem is still not investigated. In
this paper, we make the first attempt to design an interpretable audio-visual
video captioning network to discover the association between words in sentences
and audio-visual sequences. To achieve this, we propose a multimodal
convolutional neural network-based audio-visual video captioning framework and
introduce a modality-aware module for exploring modality selection during
sentence generation. Besides, we collect new audio captioning and visual
captioning datasets for further exploring the interactions between auditory and
visual modalities for high-level video understanding. Extensive experiments
demonstrate that the modality-aware module makes our model interpretable on
modality selection during sentence generation. Even with the added
interpretability, our video captioning network can still achieve comparable
performance with recent state-of-the-art methods.

本论文介绍了一个多模态卷积神经网络视频字幕框架，通过引入模态感知模块，探索了视听交互对视频理解的影响，并证明该可解释模型在情况选择时取得了可比较的性能。