We conjecture that hidden state vectors corresponding to individual input
tokens encode information sufficient to accurately predict several tokens
ahead. More concretely, in this paper we ask: Given a hidden (internal)
representation of a single token at position $t$ in an input, can we reliably
anticipate the tokens that will appear at positions $\geq t + 2$? To test this,
we measure linear approximation and causal intervention methods in GPT-J-6B to
evaluate the degree to which individual hidden states in the network contain
signal rich enough to predict future hidden states and, ultimately, token
outputs. We find that, at some layers, we can approximate a model's output with
more than 48% accuracy with respect to its prediction of subsequent tokens
through a single hidden state. Finally we present a "Future Lens" visualization
that uses these methods to create a new view of transformer states.

利用隐藏状态向量进行预测模型，使用线性逼近和因果干预方法对 GPT-J-6B 网络中的隐藏状态进行评估，发现某些层的单一隐藏状态可以以超过 48% 的准确率近似模型输出，并提出了 “未来镜头” 可视化方法来呈现 Transformer 状态。

未来视角：从单个隐藏状态预测后续标记

Future Lens: Anticipating Subsequent Tokens from a Single Hidden State

A significant number of neural architectures for reading comprehension have
recently been developed and evaluated on large cloze-style datasets. We present
experiments supporting the emergence of "predication structure" in the hidden
state vectors of these readers. More specifically, we provide evidence that the
hidden state vectors represent atomic formulas $\Phi[c]$ where $\Phi$ is a
semantic property (predicate) and $c$ is a constant symbol entity identifier.

通过对阅读理解神经网络架构进行实验，我们证实了隐藏状态向量中出现 “预测结构” 的现象，表明隐藏状态向量表示包含语义属性（谓词）和常量实体标识符的原子公式。