We conducted a survey of 135 software engineering (SE) practitioners to
understand how they use Generative AI-based chatbots like ChatGPT for SE tasks.
We find that they want to use ChatGPT for SE tasks like software library
selection but often worry about the truthfulness of ChatGPT responses. We
developed a suite of techniques and a tool called CID (ChatGPT Incorrectness
Detector) to automatically test and detect the incorrectness in ChatGPT
responses. CID is based on the iterative prompting to ChatGPT by asking it
contextually similar but textually divergent questions (using an approach that
utilizes metamorphic relationships in texts). The underlying principle in CID
is that for a given question, a response that is different from other responses
(across multiple incarnations of the question) is likely an incorrect response.
In a benchmark study of library selection, we show that CID can detect
incorrect responses from ChatGPT with an F1-score of 0.74 - 0.75.

通过调查 135 名软件工程师对 Generative AI-based chatbots 像 ChatGPT 在软件工程任务中的使用，我们发现他们都想将 ChatGPT 用于软件库选择等任务，但也常常担心 ChatGPT 回复的真实性。我们开发了一套技术和一个名为 CID（ChatGPT 错误检测器）的工具，通过询问环境上类似但内容上有差异的问题（使用利用文本中变形关系的方法），对 ChatGPT 的回复进行自动测试和检测不正确的回复。CID 的基本原则是对于一个给定的问题，与其他回复（在多个问题的不同表达中）不同的回复很可能是不正确的回复。在一个关于软件库选择的基准研究中，我们展示 CID 可以以 0.74 至 0.75 的 F1 分数来检测 ChatGPT 中的不正确回复。