Causality is fundamental in human cognition and has drawn attention in
diverse research fields. With growing volumes of textual data, discerning
causalities within text data is crucial, and causal text mining plays a pivotal
role in extracting meaningful patterns. This study conducts comprehensive
evaluations of ChatGPT's causal text mining capabilities. Firstly, we introduce
a benchmark that extends beyond general English datasets, including
domain-specific and non-English datasets. We also provide an evaluation
framework to ensure fair comparisons between ChatGPT and previous approaches.
Finally, our analysis outlines the limitations and future challenges in
employing ChatGPT for causal text mining. Specifically, our analysis reveals
that ChatGPT serves as a good starting point for various datasets. However,
when equipped with a sufficient amount of training data, previous models still
surpass ChatGPT's performance. Additionally, ChatGPT suffers from the tendency
to falsely recognize non-causal sequences as causal sequences. These issues
become even more pronounced with advanced versions of the model, such as GPT-4.
In addition, we highlight the constraints of ChatGPT in handling complex
causality types, including both intra/inter-sentential and implicit causality.
The model also faces challenges with effectively leveraging in-context learning
and domain adaptation. Our code is available on
https://github.com/retarfi/gemcausal

通过综合评估 ChatGPT 在因果文本挖掘领域的能力，揭示了其适用性、性能、限制和未来挑战。尽管 ChatGPT 在各种数据集上表现良好，但在具备足够的训练数据和处理复杂因果关系方面，先前的模型仍然优于 ChatGPT，并指出其中存在误识非因果序列和处理复杂因果关系类型方面的挑战。