Structural probes learn a linear transformation to find how dependency trees are embedded in the hidden states of language models. This simple design may not allow for full exploitation of the structure of the encoded information. Hence, to investigate the structure of the encoded information to its full extent, we incorporate non-linear structural probes. We reformulate the design of non-linear structural probes introduced by White et al. making its design simpler yet effective. We also design a visualization framework that lets us qualitatively assess how strongly two words in a sentence are connected in the predicted dependency tree. We use this technique to understand which non-linear probe variant is good at encoding syntactical information. Additionally, we also use it to qualitatively investigate the structure of dependency trees that BERT encodes in each of its layers. We find that the radial basis function (RBF) is an effective non-linear probe for the BERT model than the linear probe.

通过使用非线性结构探测器来研究编码信息的结构特点，这篇论文提出了一种简化而有效的非线性结构探测器设计，并设计了一个可视化框架来定性地评估句子中两个词之间的连接强度，并用于探索BERT在各层中编码的依赖关系树的结构。研究发现，径向基函数（RBF）比线性探测器对BERT模型更有效。

用非线性力量击打“探测性”及更多