Hierarchical policies that combine language and low-level control have been
shown to perform impressively long-horizon robotic tasks, by leveraging either
zero-shot high-level planners like pretrained language and vision-language
models (LLMs/VLMs) or models trained on annotated robotic demonstrations.
However, for complex and dexterous skills, attaining high success rates on
long-horizon tasks still represents a major challenge -- the longer the task
is, the more likely it is that some stage will fail. Can humans help the robot
to continuously improve its long-horizon task performance through intuitive and
natural feedback? In this paper, we make the following observation: high-level
policies that index into sufficiently rich and expressive low-level
language-conditioned skills can be readily supervised with human feedback in
the form of language corrections. We show that even fine-grained corrections,
such as small movements ("move a bit to the left"), can be effectively
incorporated into high-level policies, and that such corrections can be readily
obtained from humans observing the robot and making occasional suggestions.
This framework enables robots not only to rapidly adapt to real-time language
feedback, but also incorporate this feedback into an iterative training scheme
that improves the high-level policy's ability to correct errors in both
low-level execution and high-level decision-making purely from verbal feedback.
Our evaluation on real hardware shows that this leads to significant
performance improvement in long-horizon, dexterous manipulation tasks without
the need for any additional teleoperation. Videos and code are available at
this https URL

通过语言反馈不断改进高层策略，使机器人在复杂的、需要长程规划的任务中表现得更加出色，无需额外的远程操作。

喊叫您的机器人：从语言纠正中实时改进

Yell At Your Robot: Improving On-the-Fly from Language Corrections

As humans, we consistently engage in interactions with our peers and receive
feedback in the form of natural language. This language feedback allows us to
reflect on our actions, maintain appropriate behavior, and rectify our errors.
The question arises naturally: can we use language feedback to align large
language models (LLMs)? In contrast to previous research that aligns LLMs with
reward or preference data, we present the first systematic exploration of
alignment through the lens of language feedback (i.e., judgment). We commence
with an in-depth investigation of potential methods that can be adapted for
aligning LLMs with judgments, revealing that these methods are unable to fully
capitalize on the judgments. To facilitate more effective utilization of
judgments, we propose a novel framework, Contrastive Unlikelihood Training
(CUT), that allows for fine-grained inappropriate content detection and
correction based on judgments. Our offline alignment results show that, with
merely 1317 off-the-shelf judgment data, CUT (LLaMA2-13b) can beat the 175B
DaVinci003 and surpass the best baseline by 52.34 points on AlpacaEval. The
online alignment results demonstrate that CUT can align LLMs (LLaMA2-chat-13b)
in an iterative fashion using model-specific judgment data, with a steady
performance improvement from 81.09 to 91.36 points on AlpacaEval. Our analysis
further suggests that judgments exhibit greater potential than rewards for LLM
alignment and warrant future research.

我们首次通过自然语言反馈的方法探索了对齐大型语言模型的可能性，并提出了一种称为 Contrastive Unlikelihood Training (CUT) 的新框架，通过细致判定检测和修正来实现对不适当内容的改进，获得了优于基线模型的好成绩。同时，我们的分析表明判定相较于奖励在 LLM 对齐方面具有更大的潜力，值得进行进一步研究。

拒绝的原因？将语言模型与判断对齐

Reasons to Reject? Aligning Language Models with Judgments

Despite numerous successes, the field of reinforcement learning (RL) remains
far from matching the impressive generalisation power of human behaviour
learning. One possible way to help bridge this gap be to provide RL agents with
richer, more human-like feedback expressed in natural language. To investigate
this idea, we first extend BabyAI to automatically generate language feedback
from the environment dynamics and goal condition success. Then, we modify the
Decision Transformer architecture to take advantage of this additional signal.
We find that training with language feedback either in place of or in addition
to the return-to-go or goal descriptions improves agents' generalisation
performance, and that agents can benefit from feedback even when this is only
available during training, but not at inference.

通过使用自动生成的语言反馈以及改进的决策变压器架构，在强化学习中引入更丰富、更贴近人类反馈的方式可以改善智能体的泛化性能。

通过自然语言反馈在目标条件下强化学习中的利用

Is Feedback All You Need? Leveraging Natural Language Feedback in  Goal-Conditioned Reinforcement Learning

Pretrained language models often generate outputs that are not in line with
human preferences, such as harmful text or factually incorrect summaries.
Recent work approaches the above issues by learning from a simple form of human
feedback: comparisons between pairs of model-generated outputs. However,
comparison feedback only conveys limited information about human preferences.
In this paper, we introduce Imitation learning from Language Feedback (ILF), a
new approach that utilizes more informative language feedback. ILF consists of
three steps that are applied iteratively: first, conditioning the language
model on the input, an initial LM output, and feedback to generate refinements.
Second, selecting the refinement incorporating the most feedback. Third,
finetuning the language model to maximize the likelihood of the chosen
refinement given the input. We show theoretically that ILF can be viewed as
Bayesian Inference, similar to Reinforcement Learning from human feedback. We
evaluate ILF's effectiveness on a carefully-controlled toy task and a realistic
summarization task. Our experiments demonstrate that large language models
accurately incorporate feedback and that finetuning with ILF scales well with
the dataset size, even outperforming finetuning on human summaries. Learning
from both language and comparison feedback outperforms learning from each
alone, achieving human-level summarization performance.

利用语言反馈进行模仿学习 (ILF) 是一种新方法，可以有效提升大型预训练语言模型的摘要性能，并比使用对比反馈或人工摘要的方法更好。