This paper presents a new model architecture, gate with inhibition MLP
(giMLP).The gate with inhibition on CycleMLP (gi-CycleMLP) can produce equal
performance on the ImageNet classification task, and it also improves the BERT,
Roberta, and DeBERTaV3 models depending on two novel techniques. The first is
the gating MLP, where matrix multiplications between the MLP and the trunk
Attention input in further adjust models' adaptation. The second is inhibition
which inhibits or enhances the branch adjustment, and with the inhibition
levels increasing, it offers models more muscular features restriction. We show
that the giCycleMLP with a lower inhibition level can be competitive with the
original CycleMLP in terms of ImageNet classification accuracy. In addition, we
also show through a comprehensive empirical study that these techniques
significantly improve the performance of fine-tuning NLU downstream tasks. As
for the gate with inhibition MLPs on DeBERTa (giDeBERTa) fine-tuning, we find
it can achieve appealing results on most parts of NLU tasks without any extra
pretraining again. We also find that with the use of Gate With Inhibition, the
activation function should have a short and smooth negative tail, with which
the unimportant features or the features that hurt models can be moderately
inhibited. The experiments on ImageNet and twelve language downstream tasks
demonstrate the effectiveness of Gate With Inhibition, both for image
classification and for enhancing the capacity of nature language fine-tuning
without any extra pretraining.

提出一种新的模型结构 Gate with Inhibition MLP (giMLP)，利用这种方法可以通过两种新技术显著改善 BERT、Roberta 和 DeBERTaV3 模型的性能，同时可提高精度，实验表明提出的方法可以在不需额外预训练的情况下，显著改善自然语言下游任务的性能。

giMLPs: MLP 中的抑制门机制

giMLPs: Gate with Inhibition Mechanism in MLPs

This paper presents a simple MLP-like architecture, CycleMLP, which is a
versatile backbone for visual recognition and dense predictions. As compared to
modern MLP architectures, e.g., MLP-Mixer, ResMLP, and gMLP, whose
architectures are correlated to image size and thus are infeasible in object
detection and segmentation, CycleMLP has two advantages compared to modern
approaches. (1) It can cope with various image sizes. (2) It achieves linear
computational complexity to image size by using local windows. In contrast,
previous MLPs have $O(N^2)$ computations due to fully spatial connections. We
build a family of models which surpass existing MLPs and even state-of-the-art
Transformer-based models, e.g., Swin Transformer, while using fewer parameters
and FLOPs. We expand the MLP-like models' applicability, making them a
versatile backbone for dense prediction tasks. CycleMLP achieves competitive
results on object detection, instance segmentation, and semantic segmentation.
In particular, CycleMLP-Tiny outperforms Swin-Tiny by 1.3% mIoU on ADE20K
dataset with fewer FLOPs. Moreover, CycleMLP also shows excellent zero-shot
robustness on ImageNet-C dataset. Code is available at
this https URL

这篇论文介绍了一种简单的 MLP 类型架构 CycleMLP 对于视觉识别和密集预测是一种通用的骨干，优于现代 MLP 架构，可适应各种图像尺寸，能够实现与图像大小的线性计算复杂度， 用局部窗口代替全空间连接，成功地应用于物体检测、实例分割和语义分割等领域，表现优异，比当前其他已知的 MLP 和变压器模型更具优势。