Purpose: Assess whether ChatGPT 4.0 is accurate enough to perform research
evaluations on journal articles to automate this time-consuming task.
Design/methodology/approach: Test the extent to which ChatGPT-4 can assess the
quality of journal articles using a case study of the published scoring
guidelines of the UK Research Excellence Framework (REF) 2021 to create a
research evaluation ChatGPT. This was applied to 51 of my own articles and
compared against my own quality judgements. Findings: ChatGPT-4 can produce
plausible document summaries and quality evaluation rationales that match the
REF criteria. Its overall scores have weak correlations with my self-evaluation
scores of the same documents (averaging r=0.281 over 15 iterations, with 8
being statistically significantly different from 0). In contrast, the average
scores from the 15 iterations produced a statistically significant positive
correlation of 0.509. Thus, averaging scores from multiple ChatGPT-4 rounds
seems more effective than individual scores. The positive correlation may be
due to ChatGPT being able to extract the author's significance, rigour, and
originality claims from inside each paper. If my weakest articles are removed,
then the correlation with average scores (r=0.200) falls below statistical
significance, suggesting that ChatGPT struggles to make fine-grained
evaluations. Research limitations: The data is self-evaluations of a
convenience sample of articles from one academic in one field. Practical
implications: Overall, ChatGPT does not yet seem to be accurate enough to be
trusted for any formal or informal research quality evaluation tasks. Research
evaluators, including journal editors, should therefore take steps to control
its use. Originality/value: This is the first published attempt at
post-publication expert review accuracy testing for ChatGPT.

ChatGPT 4.0 在评估期刊文章上进行研究评估的准确性方面仍不足，需要进一步改进和控制使用。