In the realm of deep learning, the self-attention mechanism has substantiated
its pivotal role across a myriad of tasks, encompassing natural language
processing and computer vision. Despite achieving success across diverse
applications, the traditional self-attention mechanism primarily leverages
linear transformations for the computation of query, key, and value (QKV),
which may not invariably be the optimal choice under specific circumstances.
This paper probes into a novel methodology for QKV computation-implementing a
specially-designed neural network structure for the calculation. Utilizing a
modified Marian model, we conducted experiments on the IWSLT 2017
German-English translation task dataset and juxtaposed our method with the
conventional approach. The experimental results unveil a significant
enhancement in BLEU scores with our method. Furthermore, our approach also
manifested superiority when training the Roberta model with the Wikitext-103
dataset, reflecting a notable reduction in model perplexity compared to its
original counterpart. These experimental outcomes not only validate the
efficacy of our method but also reveal the immense potential in optimizing the
self-attention mechanism through neural network-based QKV computation, paving
the way for future research and practical applications. The source code and
implementation details for our proposed method can be accessed at
this https URL

通过神经网络计算 QKV，优化自注意力机制在深度学习中的表现，并在实验中证明了该方法的有效性和潜力。