Large Language Models (LLMs) have shown remarkable capabilities in processing
both natural and programming languages, which have enabled various applications
in software engineering, such as requirement engineering, code generation, and
software testing. However, existing code generation benchmarks do not
necessarily assess the code understanding performance of LLMs, especially for
the subtle inconsistencies that may arise between code and its semantics
described in natural language.
In this paper, we propose a novel method to systematically assess the code
understanding performance of LLMs, particularly focusing on subtle differences
between code and its descriptions, by introducing code mutations to existing
code generation datasets. Code mutations are small changes that alter the
semantics of the original code, creating a mismatch with the natural language
description. We apply different types of code mutations, such as operator
replacement and statement deletion, to generate inconsistent code-description
pairs. We then use these pairs to test the ability of LLMs to correctly detect
the inconsistencies.
We propose a new LLM testing method, called Mutation-based Consistency
Testing (MCT), and conduct a case study on the two popular LLMs, GPT-3.5 and
GPT-4, using the state-of-the-art code generation benchmark, HumanEval-X, which
consists of six programming languages (Python, C++, Java, Go, JavaScript, and
Rust). We compare the performance of the LLMs across different types of code
mutations and programming languages and analyze the results. We find that the
LLMs show significant variation in their code understanding performance and
that they have different strengths and weaknesses depending on the mutation
type and language.

我们提出了一种评估 Large Language Models（LLMs）代码理解性能的新方法，通过引入代码变异来检测 LLMs 对代码和自然语言描述之间微妙差异的能力，并在各种代码变异和编程语言上对两个常见的 LLMs 进行了案例研究，发现它们在代码理解性能上存在显著的差异。