The primary goal of continual learning (CL) task in medical image
segmentation field is to solve the "catastrophic forgetting" problem, where the
model totally forgets previously learned features when it is extended to new
categories (class-level) or tasks (task-level). Due to the privacy protection,
the historical data labels are inaccessible. Prevalent continual learning
methods primarily focus on generating pseudo-labels for old datasets to force
the model to memorize the learned features. However, the incorrect
pseudo-labels may corrupt the learned feature and lead to a new problem that
the better the model is trained on the old task, the poorer the model performs
on the new tasks. To avoid this problem, we propose a network by introducing
the data-specific Mixture of Experts (MoE) structure to handle the new tasks or
categories, ensuring that the network parameters of previous tasks are
unaffected or only minimally impacted. To further overcome the tremendous
memory costs caused by introducing additional structures, we propose a Low-Rank
strategy which significantly reduces memory cost. We validate our method on
both class-level and task-level continual learning challenges. Extensive
experiments on multiple datasets show our model outperforms all other methods.

在医学图像分割领域的持续学习任务中，我们提出了一种使用数据特定的专家组（MoE）结构解决新任务或类别的问题的网络，以确保网络参数对先前任务的影响最小化，并通过引入低秩策略显著降低引入附加结构所带来的内存开销。在多个数据集上进行的广泛实验证明了我们的模型优于其他所有方法。

低维混合专家模型在医学图像分割中的应用

Low-Rank Mixture-of-Experts for Continual Medical Image Segmentation

This project addresses the challenge of human motion prediction, a critical
area for applications such as au- tonomous vehicle movement detection. Previous
works have emphasized the need for low inference times to provide real time
performance for applications like these. Our primary objective is to critically
evaluate existing model ar- chitectures, identifying their advantages and
opportunities for improvement by replicating the state-of-the-art (SOTA)
Spatio-Temporal Transformer model as best as possible given computational con-
straints. These models have surpassed the limitations of RNN-based models and
have demonstrated the ability to generate plausible motion sequences over both
short and long term horizons through the use of spatio-temporal rep-
resentations. We also propose a novel architecture to ad- dress challenges of
real time inference speed by incorpo- rating a Mixture of Experts (MoE) block
within the Spatial- Temporal (ST) attention layer. The particular variation
that is used is Soft MoE, a fully-differentiable sparse Transformer that has
shown promising ability to enable larger model capacity at lower inference
cost. We make out code publicly available at
this https URL

本研究通过复制现有的最先进（SOTA）时空转换器模型，以尽可能符合计算限制，并批判性评估现有模型架构的优势和改进机会，以解决实时推理速度挑战，并在空间 - 时间 (ST) 关注层中引入专家混合 (MoE) 块来提高模型容量和降低推理成本。

一种基于专家混合的三维人体动作预测方法

A Mixture of Experts Approach to 3D Human Motion Prediction

Learning skills by imitation is a promising concept for the intuitive
teaching of robots. A common way to learn such skills is to learn a parametric
model by maximizing the likelihood given the demonstrations. Yet, human
demonstrations are often multi-modal, i.e., the same task is solved in multiple
ways which is a major challenge for most imitation learning methods that are
based on such a maximum likelihood (ML) objective. The ML objective forces the
model to cover all data, it prevents specialization in the context space and
can cause mode-averaging in the behavior space, leading to suboptimal or
potentially catastrophic behavior. Here, we alleviate those issues by
introducing a curriculum using a weight for each data point, allowing the model
to specialize on data it can represent while incentivizing it to cover as much
data as possible by an entropy bonus. We extend our algorithm to a Mixture of
(linear) Experts (MoE) such that the single components can specialize on local
context regions, while the MoE covers all data points. We evaluate our approach
in complex simulated and real robot control tasks and show it learns from
versatile human demonstrations and significantly outperforms current SOTA
methods. A reference implementation can be found at
this https URL

本研究提出一种基于课程学习和 MoE 的模仿学习方法，用于从人类多样化的演示中教授机器人控制技能，并在复杂机器人控制任务中显著优于当前最先进的方法。