How to evaluate Artificial General Intelligence (AGI) is a critical problem
that is discussed and unsolved for a long period. In the research of narrow AI,
this seems not a severe problem, since researchers in that field focus on some
specific problems as well as one or some aspects of cognition, and the criteria
for evaluation are explicitly defined. By contrast, an AGI agent should solve
problems that are never-encountered by both agents and developers. However,
once a developer tests and debugs the agent with a problem, the
never-encountered problem becomes the encountered problem, as a result, the
problem is solved by the developers to some extent, exploiting their
experience, rather than the agents. This conflict, as we call the trap of
developers' experience, leads to that this kind of problems is probably hard to
become an acknowledged criterion. In this paper, we propose an evaluation
method named Artificial Open World, aiming to jump out of the trap. The
intuition is that most of the experience in the actual world should not be
necessary to be applied to the artificial world, and the world should be open
in some sense, such that developers are unable to perceive the world and solve
problems by themselves before testing, though after that they are allowed to
check all the data. The world is generated in a similar way as the actual
world, and a general form of problems is proposed. A metric is proposed aiming
to quantify the progress of research. This paper describes the conceptual
design of the Artificial Open World, though the formalization and the
implementation are left to the future.

本文提出了一种名为 “人工开放世界” 的评估方法，旨在解决开发者经验陷阱问题，该方法通过创建一个人工世界，以一种类似实际世界的方式生成问题并提出度量标准来量化研究进展。