The desire and ability to seek new information strategically are fundamental
to human learning but often overlooked in current language agent development.
Using a web shopping task as an example, we show that it can be reformulated
and solved as a retrieval task without a requirement of interactive information
seeking. We then redesign the task to introduce a new role of shopper, serving
as a realistically constrained communication channel. The agents in our
proposed ChatShop task explore user preferences in open-ended conversation to
make informed decisions. Our experiments demonstrate that the proposed task can
effectively evaluate the agent's ability to explore and gradually accumulate
information through multi-turn interaction. We also show that LLM-simulated
shoppers serve as a good proxy to real human shoppers and discover similar
error patterns of agents.

我们使用网络购物任务作为示例，将其重新定义为一个信息检索任务，并引入购物者角色作为一个受限通信通道，以此来评估语言智能代理系统探索和积累信息的能力。我们的实验证明，该任务能够有效地评估代理系统通过多轮交互获取和积累信息的能力。同时，我们发现基于 LLM 模拟的购物者在错误模式方面与真实人类购物者存在相似性。