Large language models (LLMs) have exhibited remarkable performance across
various tasks in natural language processing. Nevertheless, challenges still
arise when these tasks demand domain-specific expertise and advanced analytical
skills, such as conducting research surveys on a designated topic. In this
research, we develop ResearchArena, a benchmark that measures LLM agents'
ability to conduct academic surveys, an initial step of academic research
process. Specifically, we deconstructs the surveying process into three stages
1) information discovery: locating relevant papers, 2) information selection:
assessing papers' importance to the topic, and 3) information organization:
organizing papers into meaningful structures. In particular, we establish an
offline environment comprising 12.0M full-text academic papers and 7.9K survey
papers, which evaluates agents' ability to locate supporting materials for
composing the survey on a topic, rank the located papers based on their impact,
and organize these into a hierarchical knowledge mind-map. With this benchmark,
we conduct preliminary evaluations of existing techniques and find that all
LLM-based methods under-performing when compared to basic keyword-based
retrieval techniques, highlighting substantial opportunities for future
research.

利用 ResearchArena 测量大型语言模型代理在进行学术调查的能力，将调查流程分为信息发现、信息选择和信息组织三个阶段，在离线环境中评估代理定位支持材料、对定位的论文进行排名和将其组织成层次知识思维导图的能力，根据初步评估发现大型语言模型方法与基于关键词检索技术相比表现不佳，未来研究具有重要发展机会。

ResearchArena：评估 LLMs 作为研究代理的信息收集和组织能力

ResearchArena: Benchmarking LLMs' Ability to Collect and Organize  Information as Research Agents

Pinboard on Pinterest is an emerging media to engage online social media
users, on which users post online images for specific topics. Regardless of its
significance, there is little previous work specifically to facilitate
information discovery based on pinboards. This paper proposes a novel pinboard
recommendation system for Twitter users. In order to associate contents from
the two social media platforms, we propose to use MultiLabel classification to
map Twitter user followees to pinboard topics and visual diversification to
recommend pinboards given user interested topics. A preliminary experiment on a
dataset with 2000 users validated our proposed system.

本文提出了一种新型的 Pinboard 推荐系统，使用 MultiLabel 分类将 Twitter 用户的关注者映射到 Pinboard 主题，并使用视觉多样性方法为用户推荐相关的 Pinboard，该推荐系统在拥有 2000 个用户的数据集上进行了初步实验。