Clinical language processing has received a lot of attention in recent years,
resulting in new models or methods for disease phenotyping, mortality
prediction, and other tasks. Unfortunately, many of these approaches are tested
under different experimental settings (e.g., data sources, training and testing
splits, metrics, evaluation criteria, etc.) making it difficult to compare
approaches and determine state-of-the-art. To address these issues and
facilitate reproducibility and comparison, we present the Clinical Language
Understanding Evaluation (CLUE) benchmark with a set of four clinical language
understanding tasks, standard training, development, validation and testing
sets derived from MIMIC data, as well as a software toolkit. It is our hope
that these data will enable direct comparison between approaches, improve
reproducibility, and reduce the barrier-to-entry for developing novel models or
methods for these clinical language understanding tasks.

本文介绍了 Clinical Language Understanding Evaluation (CLUE) 基准测试，它通过提供一组基于 MIMIC 数据的四个临床语言理解任务、标准的训练、开发、验证和测试集，以及一个软件工具包去解决临床语言处理面临的实验设置不一等问题，能够直接比较不同方法之间的性能差异，提高可重复性，并降低开发新模型或方法的门槛。

临床语言理解评估（CLUE）

Clinical Language Understanding Evaluation (CLUE)

In February this year Google proposed a new Transformer variant called FLASH,
which has a faster speed, lower VRAM footprint and better performance. This is
achieved by designing a performant layer named GAU (Gated Attention Unit),
which combines the Attention layer and FFN. In this paper, some implementation
details are re-analyzed both theoretically and practically. We then propose a
novel GAU-based model and pre-train it on a Chinese corpus. Results of the CLUE
benchmark show that our model achieves a dev average score of 75.02, 1% higher
than RoFormerV1 and being 45% faster, which is also competitive with
RoFormerV2.

本文重新分析了 Google 在今年 2 月提出的 Transformer 变种 FLASH 的实现细节；并在此基础上，提出一种基于 GAU 的模型，并在中文语料库上进行预训练。CLUE 基准测试结果表明，该模型的开发平均分为 75.02，在速度提高 45％的情况下，比 RoFormerV1 高 1％，并与 RoFormerV2 相当有竞争力。

GAU-based 模型的实现和实验的补充材料

Supplementary Material: Implementation and Experiments for GAU-based Model

Although pre-trained models (PLMs) have achieved remarkable improvements in a
wide range of NLP tasks, they are expensive in terms of time and resources.
This calls for the study of training more efficient models with less
computation but still ensures impressive performance. Instead of pursuing a
larger scale, we are committed to developing lightweight yet more powerful
models trained with equal or less computation and friendly to rapid deployment.
This technical report releases our pre-trained model called Mengzi, which
stands for a family of discriminative, generative, domain-specific, and
multimodal pre-trained model variants, capable of a wide range of language and
vision tasks. Compared with public Chinese PLMs, Mengzi is simple but more
powerful. Our lightweight model has achieved new state-of-the-art results on
the widely-used CLUE benchmark with our optimized pre-training and fine-tuning
techniques. Without modifying the model architecture, our model can be easily
employed as an alternative to existing PLMs. Our sources are available at
this https URL

本技术报告介绍了我们的预训练模型 Mengzi，它是一种轻量但功能强大、多模式的预训练模型，经过我们优化的预训练和微调技术，已经在广泛使用的 CLUE 基准上取得了最新的最佳结果。