Despite the growing adoption of mixed reality and interactive AI agents, it
remains challenging for these systems to generate high quality 2D/3D scenes in
unseen environments. The common practice requires deploying an AI agent to
collect large amounts of data for model training for every new task. This
process is costly, or even impossible, for many domains. In this study, we
develop an infinite agent that learns to transfer knowledge memory from general
foundation models (e.g. GPT4, DALLE) to novel domains or scenarios for scene
understanding and generation in the physical or virtual world. The heart of our
approach is an emerging mechanism, dubbed Augmented Reality with Knowledge
Inference Interaction (ArK), which leverages knowledge-memory to generate
scenes in unseen physical world and virtual reality environments. The knowledge
interactive emergent ability (Figure 1) is demonstrated as the observation
learns i) micro-action of cross-modality: in multi-modality models to collect a
large amount of relevant knowledge memory data for each interaction task (e.g.,
unseen scene understanding) from the physical reality; and ii) macro-behavior
of reality-agnostic: in mix-reality environments to improve interactions that
tailor to different characterized roles, target variables, collaborative
information, and so on. We validate the effectiveness of ArK on the scene
generation and editing tasks. We show that our ArK approach, combined with
large foundation models, significantly improves the quality of generated 2D/3D
scenes, compared to baselines, demonstrating the potential benefit of
incorporating ArK in generative AI for applications such as metaverse and
gaming simulation.

研究了一种基于知识记忆的增强现实与知识推理交互机制（ArK），利用通用的基础模型将知识记忆从一个领域转移到另一个领域，使得机器学习 agent 在未知情况下可以生成高质量 2D/3D 场景，证明了增强现实技术在深度学习中的潜力。