Self-attention mechanisms have made striking state-of-the-art (SOTA) progress
in various sequence learning tasks, standing on the multi-headed dot product
attention by attending to all the global contexts at different locations.
Through a pseudo information highway, we introduce a gated component
self-dependency units (SDU) that incorporates LSTM-styled gating units to
replenish internal semantic importance within the multi-dimensional latent
space of individual representations. The subsidiary content-based SDU gates
allow for the information flow of modulated latent embeddings through skipped
connections, leading to a clear margin of convergence speed with gradient
descent algorithms. We may unveil the role of gating mechanism to aid in the
context-based Transformer modules, with hypothesizing that SDU gates,
especially on shallow layers, could push it faster to step towards suboptimal
points during the optimization process.

通过将 LSTM 并入已有的 multi-headed dot product attention 机制中作为 gated component self-dependency units，使信息流动更加顺畅，从而提高了在序列学习任务中的性能。同时还成功阐述了其在 context-based Transformer 模型中的应用，即 SDU gates 可以加速优化过程。