AI-generated text detection plays an increasingly important role in various
fields. In this study, we developed an efficient AI-generated text detection
model based on the BERT algorithm, which provides new ideas and methods for
solving related problems. In the data preprocessing stage, a series of steps
were taken to process the text, including operations such as converting to
lowercase, word splitting, removing stop words, stemming extraction, removing
digits, and eliminating redundant spaces, to ensure data quality and accuracy.
By dividing the dataset into a training set and a test set in the ratio of 60%
and 40%, and observing the changes in the accuracy and loss values during the
training process, we found that the model performed well during the training
process. The accuracy increases steadily from the initial 94.78% to 99.72%,
while the loss value decreases from 0.261 to 0.021 and converges gradually,
which indicates that the BERT model is able to detect AI-generated text with
high accuracy and the prediction results are gradually approaching the real
classification results. Further analysis of the results of the training and
test sets reveals that in terms of loss value, the average loss of the training
set is 0.0565, while the average loss of the test set is 0.0917, showing a
slightly higher loss value. As for the accuracy, the average accuracy of the
training set reaches 98.1%, while the average accuracy of the test set is
97.71%, which is not much different from each other, indicating that the model
has good generalisation ability. In conclusion, the AI-generated text detection
model based on the BERT algorithm proposed in this study shows high accuracy
and stability in experiments, providing an effective solution for related
fields.

基于 BERT 算法的人工智能生成文本检测模型在实验中表现出高准确率和稳定性，为相关领域提供了有效的解决方案。

基于 BERT 深度学习算法的 AI 生成文本检测与分类

AI-Generated Text Detection and Classification Based on BERT Deep  Learning Algorithm

Despite impressive performance on language modelling and complex reasoning
tasks, Large Language Models (LLMs) fall short on the same tasks in uncommon
settings or with distribution shifts, exhibiting some lack of generalisation
ability. This issue has usually been alleviated by feeding more training data
into the LLM. However, this method is brittle, as the scope of tasks may not be
readily predictable or may evolve, and updating the model with new data
generally requires extensive additional training. By contrast, systems, such as
causal models, that learn abstract variables and causal relationships can
demonstrate increased robustness against changes in the distribution. One
reason for this success is the existence and use of Independent Causal
Mechanisms (ICMs) representing high-level concepts that only sparsely interact.
In this work, we apply two concepts from causality to learn ICMs within LLMs.
We develop a new LLM architecture composed of multiple sparsely interacting
language modelling modules. We introduce a routing scheme to induce
specialisation of the network into domain-specific modules. We also present a
Mutual Information minimisation objective that trains a separate module to
learn abstraction and domain-invariant mechanisms. We show that such causal
constraints can improve out-of-distribution performance on abstract and causal
reasoning tasks.

利用因果关系的两个概念在大型语言模型中学习独立因果机制，通过引入路由方案实现网络的专业化，并采用最小化互信息目标训练一个独立模块来学习抽象和领域不变机制，从而改善抽象和因果推理任务的超越分布性能。

大型语言模型能学习独立因果机制吗？

Can Large Language Models Learn Independent Causal Mechanisms?

Topic models have evolved from conventional Bayesian probabilistic models to
Neural Topic Models (NTMs) over the last two decays. Although NTMs have
achieved promising performance when trained and tested on a specific corpus,
their generalisation ability across corpora is rarely studied. In practice, we
often expect that an NTM trained on a source corpus can still produce quality
topical representation for documents in a different target corpus without
retraining. In this work, we aim to improve NTMs further so that their benefits
generalise reliably across corpora and tasks. To do so, we propose to model
similar documents by minimising their semantical distance when training NTMs.
Specifically, similar documents are created by data augmentation during
training; The semantical distance between documents is measured by the
Hierarchical Topic Transport Distance (HOTT), which computes the Optimal
Transport (OT) distance between the topical representations. Our framework can
be readily applied to most NTMs as a plug-and-play module. Extensive
experiments show that our framework significantly improves the generalisation
ability regarding neural topical representation across corpora.

提出了一种通过模型相似度来改进神经主题模型的方法，利用数据增强和层次化主题传输距离来计算文档之间的语义距离，从而显著提高神经主题模型在不同语料库中的泛化能力。

走向神经主题表示的泛化

Towards Generalising Neural Topical Representations

PAC-Bayes learning is an established framework to assess the generalisation
ability of learning algorithm during the training phase. However, it remains
challenging to know whether PAC-Bayes is useful to understand, before training,
why the output of well-known algorithms generalise well. We positively answer
this question by expanding the \emph{Wasserstein PAC-Bayes} framework, briefly
introduced in \cite{amit2022ipm}. We provide new generalisation bounds
exploiting geometric assumptions on the loss function. Using our framework, we
prove, before any training, that the output of an algorithm from
\citet{lambert2022variational} has a strong asymptotic generalisation ability.
More precisely, we show that it is possible to incorporate optimisation results
within a generalisation framework, building a bridge between PAC-Bayes and
optimisation algorithms.

在训练之前，利用 PAC-Bayes 和优化算法之间的联系，扩展 Wasserstein PAC-Bayes 框架，基于损失函数的几何假设提供新的泛化界，并证明了算法输出具有强大的渐近泛化能力。