ChatGPT is a powerful large language model (LLM) that has made remarkable
progress in natural language understanding. Nevertheless, the performance and
limitations of the model still need to be extensively evaluated. As ChatGPT
covers resources such as Wikipedia and supports natural language question
answering, it has garnered attention as a potential replacement for traditional
knowledge based question answering (KBQA) models. Complex question answering is
a challenge task of KBQA, which comprehensively tests the ability of models in
semantic parsing and reasoning. To assess the performance of ChatGPT as a
question answering system (QAS) using its own knowledge, we present a framework
that evaluates its ability to answer complex questions. Our approach involves
categorizing the potential features of complex questions and describing each
test question with multiple labels to identify combinatorial reasoning.
Following the black-box testing specifications of CheckList proposed by Ribeiro
this http URL, we develop an evaluation method to measure the functionality and
reliability of ChatGPT in reasoning for answering complex questions. We use the
proposed framework to evaluate the performance of ChatGPT in question answering
on 8 real-world KB-based CQA datasets, including 6 English and 2 multilingual
datasets, with a total of approximately 190,000 test cases. We compare the
evaluation results of ChatGPT, GPT-3.5, GPT-3, and FLAN-T5 to identify common
long-term problems in LLMs. The dataset and code are available at
this https URL.

评估使用自身知识作为问答系统的 ChatGPT 在回答复杂问题方面的能力，我们提出了一个框架来评估其功能和可靠性，并使用 CheckList 和 8 个现实世界的基于 KB 的问答数据集进行了测试。 我们发现 LLM 模型的一些常见问题。