Large transformer models powered by diverse data and model scale have
dominated natural language modeling and computer vision and pushed the frontier
of multiple AI areas. In reinforcement learning (RL), despite many efforts into
transformer-based policies, a key limitation, however, is that current
transformer-based policies cannot learn by directly combining information from
multiple sub-optimal trials. In this work, we address this issue using recently
proposed chain of hindsight to relabel experience, where we train a transformer
on a sequence of trajectory experience ascending sorted according to their
total rewards. Our method consists of relabelling target return of each
trajectory to the maximum total reward among in sequence of trajectories and
training an autoregressive model to predict actions conditioning on past
states, actions, rewards, target returns, and task completion tokens, the
resulting model, Agentic Transformer (AT), can learn to improve upon itself
both at training and test time. As we show on D4RL and ExoRL benchmarks, to the
best our knowledge, this is the first time that a simple transformer-based
model performs competitively with both temporal-difference and
imitation-learning-based approaches, even from sub-optimal data. Our Agentic
Transformer also shows a promising scaling trend that bigger models
consistently improve results.

本文利用 “chain of hindsight” 方法在强化学习中训练了一个能够直接综合多个轨迹信息的 transformer 模型，并通过在 D4RL 和 ExoRL 基准测试中的表现证明了它的竞争力和可伸缩性。

后见之链中崛起的代理变形机

Emergent Agentic Transformer from Chain of Hindsight Experience

Learning from human preferences is important for language models to be
helpful and useful for humans, and to align with human and social values. Prior
work have achieved remarkable successes by learning from human feedback to
understand and follow instructions. Nonetheless, these methods are either
founded on hand-picked model generations that are favored by human annotators,
rendering them ineffective in terms of data utilization and challenging to
apply in general, or they depend on reward functions and reinforcement
learning, which are prone to imperfect reward function and extremely
challenging to optimize. In this work, we propose a novel technique, Chain of
Hindsight, that is easy to optimize and can learn from any form of feedback,
regardless of its polarity. Our idea is inspired by how humans learn from
extensive feedback presented in the form of languages. We convert all types of
feedback into sentences, which are then used to fine-tune the model, allowing
us to take advantage of the language comprehension capabilities of language
models. We condition the model on a sequence of model generations paired with
feedback. By doing so, models are trained to generate outputs based on
feedback, and models can learn to identify and correct negative attributes or
errors. Applying our method to large language models, we observed that Chain of
Hindsight significantly surpasses previous methods in aligning language models
with human preferences. We observed significant improvements on summarization
and dialogue tasks and our approach is markedly preferred in human evaluations.

通过 Chain of Hindsight 技术，可以用各种形式的反馈来 fine-tuning 语言模型，从而使得它们可以更好地满足人类的偏好，具有更好的性能。