We propose a novel approach for training large language models (LLMs) to
adhere to objectives defined within a latent embedding space. Our method
leverages reinforcement learning (RL), treating a pre-trained LLM as an
environment. Our embedding-aligned guided language (EAGLE) agent is trained to
iteratively steer the LLM's generation towards optimal regions of the latent
embedding space, w.r.t. some predefined criterion. We demonstrate the
effectiveness of the EAGLE agent using the MovieLens 25M dataset to surface
content gaps that satisfy latent user demand. We also demonstrate the benefit
of using an optimal design of a state-dependent action set to improve EAGLE's
efficiency. Our work paves the way for controlled and grounded text generation
using LLMs, ensuring consistency with domain-specific knowledge and data
representations.

我们提出了一种新颖的方法，通过在潜在嵌入空间中定义目标，对大型语言模型（LLMs）进行训练。我们的方法利用强化学习（RL），将预训练的 LLM 作为环境。我们训练的嵌入对齐引导语言（EAGLE）代理程序通过迭代地引导 LLM 的生成，使其指向潜在嵌入空间的最佳区域，相对于某些预定义的准则。我们使用 MovieLens 25M 数据集展示了 EAGLE 代理程序的有效性，以满足潜在用户需求的内容间隙。我们还展示了使用状态相关动作集的最佳设计来提高 EAGLE 效率的好处。我们的工作为使用 LLMs 进行受控和基于实际的文本生成铺平了道路，确保与领域特定知识和数据表示的一致性。

嵌入对齐的语言模型

Embedding-Aligned Language Models

Motivated by the intuitive understanding humans have about the space of
possible interactions, and the ease with which they can generalize this
understanding to previously unseen scenes, we develop an approach for learning
visual affordances for guiding robot exploration. Given an input image of a
scene, we infer a distribution over plausible future states that can be
achieved via interactions with it. We use a Transformer-based model to learn a
conditional distribution in the latent embedding space of a VQ-VAE and show
that these models can be trained using large-scale and diverse passive data,
and that the learned models exhibit compositional generalization to diverse
objects beyond the training distribution. We show how the trained affordance
model can be used for guiding exploration by acting as a goal-sampling
distribution, during visual goal-conditioned policy learning in robotic
manipulation.

通过学习视觉契合来指导机器人探索，并使用基于 Transformer 的模型来学习 VQ-VAE 潜在嵌入空间中条件分布，从而展现模型的组合泛化能力。训练完成的契合模型可用于机器人操作策略学习中的目标抽样分布。