This work proposes a training-free approach for the detection of
LLMs-generated codes, mitigating the risks associated with their indiscriminate
usage. To the best of our knowledge, our research is the first to investigate
zero-shot detection techniques applied to code generated by advanced black-box
LLMs like ChatGPT. Firstly, we find that existing training-based or zero-shot
text detectors are ineffective in detecting code, likely due to the unique
statistical properties found in code structures. We then modify the previous
zero-shot text detection method, DetectGPT (Mitchell et al., 2023) by utilizing
a surrogate white-box model to estimate the probability of the rightmost
tokens, allowing us to identify code snippets generated by language models.
Through extensive experiments conducted on the python codes of the CodeContest
and APPS dataset, our approach demonstrates its effectiveness by achieving
state-of-the-art detection results on text-davinci-003, GPT-3.5, and GPT-4
models. Moreover, our method exhibits robustness against revision attacks and
generalizes well to Java codes. We also find that the smaller code language
model like PolyCoder-160M performs as a universal code detector, outperforming
the billion-scale counterpart. The codes will be available at
this https URL Xianjun-Yang/Code_detection.git

提出了一种对 LLMs 生成的代码进行训练无关的检测方法，以减少与它们的无差别使用相关的风险。通过对现有的基于训练或零样本的文本检测方法进行研究，我们发现这些方法对代码的检测不够有效，可能是由于代码结构中所具有的独特统计特性。本研究通过修改之前的零样本文本检测方法，利用代理的白盒模型来估计最右侧标记的概率，从而能够识别由语言模型生成的代码片段。通过对 CodeContest 和 APPS 数据集的 Python 代码进行大量实验，我们的方法在 text-davinci-003、GPT-3.5 和 GPT-4 模型上取得了最新的检测结果，并且对修订攻击具有鲁棒性，并且对 Java 代码具有很好的通用性。同时，我们还发现较小的代码语言模型 PolyCoder-160M 作为通用代码检测器的表现优于十亿级别的对应模型。代码可在此 https URL Xianjun-Yang/Code_detection.git 获取。