The Mixture-of-Experts (MoE) approach has demonstrated outstanding
scalability in multi-task learning including low-level upstream tasks such as
concurrent removal of multiple adverse weather effects. However, the
conventional MoE architecture with parallel Feed Forward Network (FFN) experts
leads to significant parameter and computational overheads that hinder its
efficient deployment. In addition, the naive MoE linear router is suboptimal in
assigning task-specific features to multiple experts which limits its further
scalability. In this work, we propose an efficient MoE architecture with weight
sharing across the experts. Inspired by the idea of linear feature modulation
(FM), our architecture implicitly instantiates multiple experts via learnable
activation modulations on a single shared expert block. The proposed Feature
Modulated Expert (FME) serves as a building block for the novel
Mixture-of-Feature-Modulation-Experts (MoFME) architecture, which can scale up
the number of experts with low overhead. We further propose an
Uncertainty-aware Router (UaR) to assign task-specific features to different FM
modules with well-calibrated weights. This enables MoFME to effectively learn
diverse expert functions for multiple tasks. The conducted experiments on the
multi-deweather task show that our MoFME outperforms the baselines in the image
restoration quality by 0.1-0.2 dB and achieves SOTA-compatible performance
while saving more than 72% of parameters and 39% inference time over the
conventional MoE counterpart. Experiments on the downstream segmentation and
classification tasks further demonstrate the generalizability of MoFME to real
open-world applications.

我们提出了一个高效的 Mixture-of-Feature-Modulation-Experts（MoFME）架构，通过权重共享在一组专家中隐式地实例化多个专家。同时，我们还提出了一种具有良好校准权重的不确定性感知路由器（UaR），以将任务特定的特征分配给不同的 FM 模块，从而使 MoFME 能够有效地学习多个任务的不同专家功能。在多去污任务上进行的实验证明，我们的 MoFME 在图像恢复质量方面优于基准 0.1-0.2 dB，并且相比传统的 MoE 方法，参数节省超过 72％，推理时间节省 39％以上。在下游分割和分类任务上进行的实验进一步证明了 MoFME 在真实开放世界应用中的普适性。