In recent years, Transformer-based models such as the Switch Transformer have
achieved remarkable results in natural language processing tasks. However,
these models are often too complex and require extensive pre-training, which
limits their effectiveness for small clinical text classification tasks with
limited data. In this study, we propose a simplified Switch Transformer
framework and train it from scratch on a small French clinical text
classification dataset at CHU Sainte-Justine hospital. Our results demonstrate
that the simplified small-scale Transformer models outperform pre-trained
BERT-based models, including DistillBERT, CamemBERT, FlauBERT, and FrALBERT.
Additionally, using a mixture of expert mechanisms from the Switch Transformer
helps capture diverse patterns; hence, the proposed approach achieves better
results than a conventional Transformer with the self-attention mechanism.
Finally, our proposed framework achieves an accuracy of 87\%, precision at
87\%, and recall at 85\%, compared to the third-best pre-trained BERT-based
model, FlauBERT, which achieved an accuracy of 84\%, precision at 84\%, and
recall at 84\%. However, Switch Transformers have limitations, including a
generalization gap and sharp minima. We compare it with a multi-layer
perceptron neural network for small French clinical narratives classification
and show that the latter outperforms all other models.

本研究提出了一种简化的 Switch Transformer 架构，针对小型法国临床文本分类数据集进行 scratch 训练。结果表明，简化的小型 Transformer 模型的性能优于包括 DistillBERT、CamemBERT、FlauBERT 和 FrALBERT 在内的预训练 BERT 模型。

临床叙述分类的小型开关变压器和基于 NLP 的模型

A Small-Scale Switch Transformer and NLP-based Model for Clinical Narratives Classification

In deep learning, models typically reuse the same parameters for all inputs.
Mixture of Experts (MoE) defies this and instead selects different parameters
for each incoming example. The result is a sparsely-activated model -- with
outrageous numbers of parameters -- but a constant computational cost. However,
despite several notable successes of MoE, widespread adoption has been hindered
by complexity, communication costs and training instability -- we address these
with the Switch Transformer. We simplify the MoE routing algorithm and design
intuitive improved models with reduced communication and computational costs.
Our proposed training techniques help wrangle the instabilities and we show
large sparse models may be trained, for the first time, with lower precision
(bfloat16) formats. We design models based off T5-Base and T5-Large to obtain
up to 7x increases in pre-training speed with the same computational resources.
These improvements extend into multilingual settings where we measure gains
over the mT5-Base version across all 101 languages. Finally, we advance the
current scale of language models by pre-training up to trillion parameter
models on the "Colossal Clean Crawled Corpus" and achieve a 4x speedup over the
T5-XXL model.

通过 Switch Transformer 和降低精度格式，本文展示了如何简化 MoE 路由算法，降低信息交流和计算成本，以及解决训练不稳定问题，并在多语言领域提高了预训练速度并推动了现有语言模型的规模扩大，以 trillion 参数为例，实现了超过 4 倍的速度提升。