Large Language Models (LLMs) have demonstrated impressive capabilities across
a range of scientific tasks including mathematics, physics, and chemistry.
Despite their successes, the effectiveness of LLMs in handling complex
statistical tasks remains systematically under-explored. To bridge this gap, we
introduce StatQA, a new benchmark designed for statistical analysis tasks.
StatQA comprises 11,623 examples tailored to evaluate LLMs' proficiency in
specialized statistical tasks and their applicability assessment capabilities,
particularly for hypothesis testing methods. We systematically experiment with
representative LLMs using various prompting strategies and show that even
state-of-the-art models such as GPT-4o achieve a best performance of only
64.83%, indicating significant room for improvement. Notably, while open-source
LLMs (e.g. LLaMA-3) show limited capability, those fine-tuned ones exhibit
marked improvements, outperforming all in-context learning-based methods (e.g.
GPT-4o). Moreover, our comparative human experiments highlight a striking
contrast in error types between LLMs and humans: LLMs primarily make
applicability errors, whereas humans mostly make statistical task confusion
errors. This divergence highlights distinct areas of proficiency and
deficiency, suggesting that combining LLM and human expertise could lead to
complementary strengths, inviting further investigation into their
collaborative potential.

LLMs 在处理复杂统计任务方面存在显著改进空间，引入 StatQA 作为新的基准测试以评估 LLMs 在特定统计任务和应用评估能力方面的表现，并突显 LLMs 和人类在错误类型上的差异，表明结合 LLMs 和人类专业知识有助于相互补充优势，进一步探索其合作潜力。