Prior work has shown the existence of contextual neurons in language models,
including a neuron that activates on German text. We show that this neuron
exists within a broader contextual n-gram circuit: we find late layer neurons
which recognize and continue n-grams common in German text, but which only
activate if the German neuron is active. We investigate the formation of this
circuit throughout training and find that it is an example of what we call a
second-order circuit. In particular, both the constituent n-gram circuits and
the German detection circuit which culminates in the German neuron form with
independent functions early in training - the German detection circuit
partially through modeling German unigram statistics, and the n-grams by
boosting appropriate completions. Only after both circuits have already formed
do they fit together into a second-order circuit. Contrary to the hypotheses
presented in prior work, we find that the contextual n-gram circuit forms
gradually rather than in a sudden phase transition. We further present a range
of anomalous observations such as a simultaneous phase transition in many tasks
coinciding with the learning rate warm-up, and evidence that many context
neurons form simultaneously early in training but are later unlearned.

语境性神经元与 n-gram 电路构成的二阶电路在语言模型中展现渐进形成的过程，其中的德文检测电路通过对德文单一字统计建模的方式形成，而 n-grams 电路则通过推动合适的完整性来形成。

语言模型中上下文 N-Gram 的训练动态

Training Dynamics of Contextual N-Grams in Language Models

Text readability assessment has a wide range of applications for different
target people, from language learners to people with disabilities. The fast
pace of textual content production on the web makes it impossible to measure
text complexity without the benefit of machine learning and natural language
processing techniques. Although various research addressed the readability
assessment of English text in recent years, there is still room for improvement
of the models for other languages. In this paper, we proposed a new model for
text complexity assessment for German text based on transfer learning. Our
results show that the model outperforms more classical solutions based on
linguistic features extraction from input text. The best model is based on the
BERT pre-trained language model achieved the Root Mean Square Error (RMSE) of
0.483.

本研究基于迁移学习提出了一种新的德文文本复杂度评估模型，结果表明该模型优于传统的基于文本语言特征提取的解决方案，最佳模型使用 BERT 预训练语言模型取得了 0.483 的均方根误差。