We introduceDropDim, a structured dropout method designed for regularizing
the self-attention mechanism, which is a key component of the transformer. In
contrast to the general dropout method, which randomly drops neurons, DropDim
drops part of the embedding dimensions. In this way, the semantic information
can be completely discarded. Thus, the excessive coadapting between different
embedding dimensions can be broken, and the self-attention is forced to encode
meaningful featureswith a certain number of embedding dimensions erased.
Experiments on a wide range of tasks executed on the MUST-C English-Germany
dataset show that DropDim can effectively improve model performance, reduce
over-fitting, and show complementary effects with other regularization methods.
When combined with label smoothing, the WER can be reduced from 19.1% to 15.1%
on the ASR task, and the BLEU value can be increased from26.90 to 28.38 on the
MT task. On the ST task, the model can reach a BLEU score of 22.99, an increase
by 1.86 BLEU points compared to the strong baseline.

介绍了 DropDim，一种用于规范 Transformer 中关键组件 —— 自我注意力机制的结构化丢弃方法。与通用的随机丢弃方法不同，DropDim 会丢弃嵌入维度的一部分，从而避免了嵌入维度间的过多协同适应。实验证明，DropDim 能够有效地提高模型性能，在防止过拟合和与其他规范方法互补方面表现出色。