Transformers have revolutionized various domains of artificial intelligence
due to their unique ability to model long-range dependencies in data. However,
they lack in nuanced, context-dependent modulation of features and information
flow. This paper introduces two significant enhancements to the transformer
architecture - the Evaluator Adjuster Unit (EAU) and Gated Residual Connections
(GRC) - designed to address these limitations. The EAU dynamically modulates
attention outputs based on the relevance of the input context, allowing for
more adaptive response patterns. Concurrently, the GRC modifies the
transformer's residual connections through a gating mechanism that selectively
controls the information flow, thereby enhancing the network's ability to focus
on contextually important features. We evaluate the performance of these
enhancements across several benchmarks in natural language processing. Our
results demonstrate improved adaptability and efficiency, suggesting that these
modifications could set new standards for designing flexible and context-aware
transformer models.

由于其在数据中建模长距离依赖的独特能力，transformers 已经彻底改变了人工智能的各个领域。然而，它们在特征和信息流的细微变化上存在不足。本文引入了两个重要的改进来解决 transformer 架构的这些局限性 - 评估者调整单元 (EAU) 和门控残差连接 (GRC)。EAU 根据输入上下文的相关性动态调节注意力输出，从而允许更具适应性的响应模式。同时，GRC 通过门控机制修改 transformer 的残差连接，有选择地控制信息流动，从而增强网络集中关注上下文重要特征的能力。我们在自然语言处理的多个基准测试上评估了这些改进的性能。我们的结果表明，这些改动改善了适应性和效率，可能会为设计灵活和上下文感知的 transformer 模型设定新的标准。