Large language models (LLMs) have exhibited remarkable performance across
various tasks in natural language processing. Nevertheless, challenges still
arise when these tasks demand domain-specific expertise and advanced analytical
skills, such as conducting research surveys on a designated topic. In this
research, we develop ResearchArena, a benchmark that measures LLM agents'
ability to conduct academic surveys, an initial step of academic research
process. Specifically, we deconstructs the surveying process into three stages
1) information discovery: locating relevant papers, 2) information selection:
assessing papers' importance to the topic, and 3) information organization:
organizing papers into meaningful structures. In particular, we establish an
offline environment comprising 12.0M full-text academic papers and 7.9K survey
papers, which evaluates agents' ability to locate supporting materials for
composing the survey on a topic, rank the located papers based on their impact,
and organize these into a hierarchical knowledge mind-map. With this benchmark,
we conduct preliminary evaluations of existing techniques and find that all
LLM-based methods under-performing when compared to basic keyword-based
retrieval techniques, highlighting substantial opportunities for future
research.

利用 ResearchArena 测量大型语言模型代理在进行学术调查的能力，将调查流程分为信息发现、信息选择和信息组织三个阶段，在离线环境中评估代理定位支持材料、对定位的论文进行排名和将其组织成层次知识思维导图的能力，根据初步评估发现大型语言模型方法与基于关键词检索技术相比表现不佳，未来研究具有重要发展机会。