The rapid advancements in Large Language Models (LLMs) have opened new
avenues for automating complex tasks in AI research. This paper investigates
the efficacy of different LLMs-Mistral 7B, Llama-2, GPT-4-Turbo and GPT-4.o in
extracting leaderboard information from empirical AI research articles. We
explore three types of contextual inputs to the models: DocTAET (Document
Title, Abstract, Experimental Setup, and Tabular Information), DocREC (Results,
Experiments, and Conclusions), and DocFULL (entire document). Our comprehensive
study evaluates the performance of these models in generating (Task, Dataset,
Metric, Score) quadruples from research papers. The findings reveal significant
insights into the strengths and limitations of each model and context type,
providing valuable guidance for future AI research automation efforts.

该研究通过评估 Mistral 7B、Llama-2、GPT-4-Turbo 和 GPT-4.o 这四个 LLM 模型对实证 AI 研究文章中的领先者信息的提取效果，探讨了三种上下文输入类型（DocTAET、DocREC 和 DocFULL）对模型的影响，综合评估了这些模型在从研究论文中生成（任务、数据集、指标、得分）四元组方面的性能，揭示了每个模型和上下文类型的优势和局限性，为未来的 AI 研究自动化工作提供了有价值的指导。