This paper mainly describes the dma submission to the TempoWiC task, which
achieves a macro-F1 score of 77.05% and attains the first place in this task.
We first explore the impact of different pre-trained language models. Then we
adopt data cleaning, data augmentation, and adversarial training strategies to
enhance the model generalization and robustness. For further improvement, we
integrate POS information and word semantic representation using a
Mixture-of-Experts (MoE) approach. The experimental results show that MoE can
overcome the feature overuse issue and combine the context, POS, and word
semantic features well. Additionally, we use a model ensemble method for the
final prediction, which has been proven effective by many research works.

本文主要描述了 DMA 提交至 TempoWiC 任务的情况，其中取得了 77.05％的宏 F1 得分，获得了该任务的第一名。我们首先探讨了不同预训练语言模型的影响。然后，我们采用数据清洗，数据增强和对抗训练策略来增强模型的泛化能力和鲁棒性。为了进一步提高性能，我们使用混合专家 (MoE) 方法集成 POS 信息和词义表示。实验结果表明，MoE 可以克服特征过度使用的问题，并很好地组合上下文，POS 和词义特征。此外，我们还使用了模型集成方法进行最终预测，这已被许多研究工作证明是有效的。

利用深度多元专家方法检测 TempoWiC 中的词义漂移

Using Deep Mixture-of-Experts to Detect Word Meaning Shift for TempoWiC

Vision transformers have shown excellent performance in computer vision
tasks. As the computation cost of their self-attention mechanism is expensive,
recent works tried to replace the self-attention mechanism in vision
transformers with convolutional operations, which is more efficient with
built-in inductive bias. However, these efforts either ignore multi-level
features or lack dynamic prosperity, leading to sub-optimal performance. In
this paper, we propose a Dynamic Multi-level Attention mechanism (DMA), which
captures different patterns of input images by multiple kernel sizes and
enables input-adaptive weights with a gating mechanism. Based on DMA, we
present an efficient backbone network named DMFormer. DMFormer adopts the
overall architecture of vision transformers, while replacing the self-attention
mechanism with our proposed DMA. Extensive experimental results on ImageNet-1K
and ADE20K datasets demonstrated that DMFormer achieves state-of-the-art
performance, which outperforms similar-sized vision transformers(ViTs) and
convolutional neural networks (CNNs).

本文提出了一种动态多级注意力机制 (DMA)，它通过多个卷积核大小捕捉输入图像的不同模式，并通过门控机制实现输入自适应权重，然后提出了一种名为 DMFormer 的有效骨干网络，该网络采用了 DMA 替代了视觉变换器中的自我关注机制。在 ImageNet-1K 和 ADE20K 数据集上的广泛实验结果表明，DMFormer 具有先进的性能，优于大小相似的视觉变压器 (ViTs) 和卷积神经网络 (CNNs)。