Structural probes learn a linear transformation to find how dependency trees
are embedded in the hidden states of language models. This simple design may
not allow for full exploitation of the structure of the encoded information.
Hence, to investigate the structure of the encoded information to its full
extent, we incorporate non-linear structural probes. We reformulate the design
of non-linear structural probes introduced by White et al. making its design
simpler yet effective. We also design a visualization framework that lets us
qualitatively assess how strongly two words in a sentence are connected in the
predicted dependency tree. We use this technique to understand which non-linear
probe variant is good at encoding syntactical information. Additionally, we
also use it to qualitatively investigate the structure of dependency trees that
BERT encodes in each of its layers. We find that the radial basis function
(RBF) is an effective non-linear probe for the BERT model than the linear
probe.

通过使用非线性结构探测器来研究编码信息的结构特点，这篇论文提出了一种简化而有效的非线性结构探测器设计，并设计了一个可视化框架来定性地评估句子中两个词之间的连接强度，并用于探索 BERT 在各层中编码的依赖关系树的结构。研究发现，径向基函数（RBF）比线性探测器对 BERT 模型更有效。