In modern machine (ML) learning systems, Transformer-based architectures have
achieved milestone success across a broad spectrum of tasks, yet understanding
their operational mechanisms remains an open problem. To improve the
transparency of ML systems, automata extraction methods, which interpret
stateful ML models as automata typically through formal languages, have proven
effective for explaining the mechanism of recurrent neural networks (RNNs).
However, few works have been applied to this paradigm to Transformer models. In
particular, understanding their processing of formal languages and identifying
their limitations in this area remains unexplored. In this paper, we propose an
automata extraction algorithm specifically designed for Transformer models.
Treating the Transformer model as a black-box system, we track the model
through the transformation process of their internal latent representations
during their operations, and then use classical pedagogical approaches like L*
algorithm to interpret them as deterministic finite-state automata (DFA).
Overall, our study reveals how the Transformer model comprehends the structure
of formal languages, which not only enhances the interpretability of the
Transformer-based ML systems but also marks a crucial step toward a deeper
understanding of how ML systems process formal languages. Code and data are
available at this https URL

我们提出了一种专门为 Transformer 模型设计的自动机提取算法，通过对 Transformer 模型的内部潜在表示进行转换过程的跟踪并使用经典的教学方法来将其解释为确定有限状态自动机（DFA），揭示了 Transformer 模型如何理解形式语言的结构，从而增强了 Transformer 模型的可解释性并深入了解了机器学习系统如何处理形式语言。