While Large Language Models (LLMs) have demonstrated significant promise as
agents in interactive tasks, their substantial computational requirements and
restricted number of calls constrain their practical utility, especially in
long-horizon interactive tasks such as decision-making or in scenarios
involving continuous ongoing tasks. To address these constraints, we propose a
method for transferring the performance of an LLM with billions of parameters
to a much smaller language model (770M parameters). Our approach involves
constructing a hierarchical agent comprising a planning module, which learns
through Knowledge Distillation from an LLM to generate sub-goals, and an
execution module, which learns to accomplish these sub-goals using elementary
actions. In detail, we leverage an LLM to annotate an oracle path with a
sequence of sub-goals towards completing a goal. Subsequently, we utilize this
annotated data to fine-tune both the planning and execution modules.
Importantly, neither module relies on real-time access to an LLM during
inference, significantly reducing the overall cost associated with LLM
interactions to a fixed cost. In ScienceWorld, a challenging and multi-task
interactive text environment, our method surpasses standard imitation learning
based solely on elementary actions by 16.7% (absolute). Our analysis highlights
the efficiency of our approach compared to other LLM-based methods. Our code
and annotated data for distillation can be found on GitHub.

通过层次化代理模型和知识蒸馏，将大型语言模型的性能通过转移学习到小型语言模型，在科学世界的多任务交互文本环境中胜过其他大型语言模型方法，从而提高了效率。

子目标蒸馏：提升小型语言代理的方法

Sub-goal Distillation: A Method to Improve Small Language Agents

Solving long-horizon, temporally-extended tasks using Reinforcement Learning
(RL) is challenging, compounded by the common practice of learning without
prior knowledge (or tabula rasa learning). Humans can generate and execute
plans with temporally-extended actions and quickly learn to perform new tasks
because we almost never solve problems from scratch. We want autonomous agents
to have this same ability. Recently, LLMs have been shown to encode a
tremendous amount of knowledge about the world and to perform impressive
in-context learning and reasoning. However, using LLMs to solve real world
problems is hard because they are not grounded in the current task. In this
paper we exploit the planning capabilities of LLMs while using RL to provide
learning from the environment, resulting in a hierarchical agent that uses LLMs
to solve long-horizon tasks. Instead of completely relying on LLMs, they guide
a high-level policy, making learning significantly more sample efficient. This
approach is evaluated in simulation environments such as MiniGrid, SkillHack,
and Crafter, and on a real robot arm in block manipulation tasks. We show that
agents trained using our approach outperform other baselines methods and, once
trained, don't need access to LLMs during deployment.

利用强化学习解决长期和拓展性任务很具挑战性，特别是在没有先验知识的情况下，为了提高样本效率，本文通过利用 LLMs 的规划能力结合强化学习的环境学习，构建了一个层次化代理，用于解决长期任务，并在 MiniGrid、SkillHack 和 Crafter 等仿真环境以及实际机械臂的块操作任务中验证了该方法的优越性能，且训练完成后不需要依赖 LLMs 进行部署。

LLM 增强分层代理

LLM Augmented Hierarchical Agents

Machine learning has the potential to automate molecular design and
drastically accelerate the discovery of new functional compounds. Towards this
goal, generative models and reinforcement learning (RL) using string and graph
representations have been successfully used to search for novel molecules.
However, these approaches are limited since their representations ignore the
three-dimensional (3D) structure of molecules. In fact, geometry plays an
important role in many applications in inverse molecular design, especially in
drug discovery. Thus, it is important to build models that can generate
molecular structures in 3D space based on property-oriented geometric
constraints. To address this, one approach is to generate molecules as 3D point
clouds by sequentially placing atoms at locations in space -- this allows the
process to be guided by physical quantities such as energy or other properties.
However, this approach is inefficient as placing individual atoms makes the
exploration unnecessarily deep, limiting the complexity of molecules that can
be generated. Moreover, when optimizing a molecule, organic and medicinal
chemists use known fragments and functional groups, not single atoms. We
introduce a novel RL framework for scalable 3D design that uses a hierarchical
agent to build molecules by placing molecular substructures sequentially in 3D
space, thus attempting to build on the existing human knowledge in the field of
molecular design. In a variety of experiments with different substructures, we
show that our agent, guided only by energy considerations, can efficiently
learn to produce molecules with over 100 atoms from many distributions
including drug-like molecules, organic LED molecules, and biomolecules.

本文提出了一种基于强化学习框架的分层代理方法，能够在三维空间中逐步放置分子亚结构，从而高效地学习如何建造具有不同分布的分子，包括药物样分子、有机发光二极管分子和生物分子，只利用能量考虑即可。