Although it is known that transformer language models (LMs) pass features
from early layers to later layers, it is not well understood how this
information is represented and routed by the model. By analyzing particular
mechanism LMs use to accomplish this, we find that it is also used to recall
items from a list, and show that this mechanism can explain an otherwise
arbitrary-seeming sensitivity of the model to the order of items in the prompt.
Specifically, we find that models write into low-rank subspaces of the residual
stream to represent features which are then read out by specific later layers,
forming low-rank communication channels between layers. By decomposing
attention head weight matrices with the Singular Value Decomposition (SVD), we
find that previously described interactions between heads separated by one or
more layers can be predicted via analysis of their weight matrices. We show
that it is possible to manipulate the internal model representations as well as
edit model weights based on the mechanism we discover in order to significantly
improve performance on our synthetic Laundry List task, which requires recall
from a list, often improving task accuracy by over 20%. Our analysis reveals a
surprisingly intricate interpretable structure learned from language model
pretraining, and helps us understand why sophisticated LMs sometimes fail in
simple domains, facilitating future analysis of more complex behaviors.

通过分析 Transformer 语言模型中的机制，研究揭示了其通过低秩通信信道在不同层之间传递和存储特征的方式，通过分解注意力头权重矩阵进而预测层间相互作用的方法，以及利用该机制改善内部模型表示和权重来提升任务性能的结构学习，为进一步分析复杂行为提供帮助。

对 Transformer 语言模型的层间通信的理解

Talking Heads: Understanding Inter-layer Communication in Transformer  Language Models

Transformer-based models have achieved great breakthroughs in recent years.
However, there are many significant questions that have not been answered in
the field of explaining the reason why the models have powerful outputs. We do
not know how to locate the models' important parameters storing the knowledge
for predicting the next word, and whether these parameters are stored on the
same layer/module or different ones. Moreover, we do not understand the
mechanism to merge the knowledge into the final embedding for next word
prediction. In this paper, we explore the residual stream of transformers to
increase the interpretability. We find the mechanism behind residual connection
is a direct addition function on before-softmax values, so the probabilities of
tokens with larger before-softmax values will increase. Moreover, we prove that
using log probability increase as contribution scores is reasonable, and based
on this we can locate important parameters. Besides, we propose a method to
analyze how previous layers affect upper layers by comparing the inner
products. The experimental results and case study show that our research can
increase the interpretability of transformer-based models. We will release our
code on this https URL

在本文中，我们通过研究 Transformer 模型的残差流来增强其可解释性。我们发现残差连接的机制是在 softmax 之前的值上进行直接相加，从而使得具有更大 softmax 之前值的标记的概率增加。此外，我们证明了使用对数概率增加作为贡献分数是合理的，并基于此定位重要参数。通过比较内积，我们还提出了一种分析前一层对上层影响的方法。实验结果和案例研究表明，我们的研究可以增强基于 Transformer 的模型的可解释性。我们将在此 https URL 上发布我们的代码。

探索 Transformer 的残差流

Exploring the Residual Stream of Transformers

With the rapid evolution of AI Generated Content (AIGC), forged images
produced through this technology are inherently more deceptive and require less
human intervention compared to traditional Computer-generated Graphics (CG).
However, owing to the disparities between CG and AIGC, conventional CG
detection methods tend to be inadequate in identifying AIGC-produced images. To
address this issue, our research concentrates on the text-to-image generation
process in AIGC. Initially, we first assemble two text-to-image databases
utilizing two distinct AI systems, DALLE2 and DreamStudio. Aiming to
holistically capture the inherent anomalies produced by AIGC, we develope a
robust dual-stream network comprised of a residual stream and a content stream.
The former employs the Spatial Rich Model (SRM) to meticulously extract various
texture information from images, while the latter seeks to capture additional
forged traces in low frequency, thereby extracting complementary information
that the residual stream may overlook. To enhance the information exchange
between these two streams, we incorporate a cross multi-head attention
mechanism. Numerous comparative experiments are performed on both databases,
and the results show that our detection method consistently outperforms
traditional CG detection techniques across a range of image resolutions.
Moreover, our method exhibits superior performance through a series of
robustness tests and cross-database experiments. When applied to widely
recognized traditional CG benchmarks such as SPL2018 and DsTok, our approach
significantly exceeds the capabilities of other existing methods in the field
of CG detection.

本研究提出了一种使用文本到图像生成过程中的残差网络和内容网络结合的检测方法以及跨多头注意机制来检测 AI 产生的虚假图片，并取得了优于传统 CG 检测技术的结果。