Training large transformers using next-token prediction has given rise to
groundbreaking advancements in AI. While this generative AI approach has
produced impressive results, it heavily leans on human supervision. Even
state-of-the-art AI models like ChatGPT depend on fine-tuning through human
demonstrations, demanding extensive human input and domain expertise. This
strong reliance on human oversight poses a significant hurdle to the
advancement of AI innovation. To address this limitation, we propose a novel
paradigm termed Exploratory AI (EAI) aimed at autonomously generating
high-quality training data. Drawing inspiration from unsupervised reinforcement
learning (RL) pretraining, EAI achieves exploration within the natural language
space. We accomplish this by harnessing large language models to assess the
novelty of generated content. Our approach employs two key components: an actor
that generates novel content following exploration principles and a critic that
evaluates the generated content, offering critiques to guide the actor.
Empirical evaluations demonstrate that EAI significantly boosts model
performance on complex reasoning tasks, addressing the limitations of
human-intensive supervision.

使用下一个令牌预测训练大型 Transformer 在人工智能方面取得了突破性进展，但这种生成式人工智能方法严重依赖人类监督，我们提出了一种新的自主生成高质量训练数据的范式 —— 探索型人工智能 (EAI)，通过利用大型语言模型在自然语言空间内进行探索达到这一目的。实证评估表明，EAI 显著提升了模型在复杂推理任务上的性能，解决了人类密集监督的局限。