Policies often fail due to distribution shift -- changes in the state and
reward that occur when a policy is deployed in new environments. Data
augmentation can increase robustness by making the model invariant to
task-irrelevant changes in the agent's observation. However, designers don't
know which concepts are irrelevant a priori, especially when different end
users have different preferences about how the task is performed. We propose an
interactive framework to leverage feedback directly from the user to identify
personalized task-irrelevant concepts. Our key idea is to generate
counterfactual demonstrations that allow users to quickly identify possible
task-relevant and irrelevant concepts. The knowledge of task-irrelevant
concepts is then used to perform data augmentation and thus obtain a policy
adapted to personalized user objectives. We present experiments validating our
framework on discrete and continuous control tasks with real human users. Our
method (1) enables users to better understand agent failure, (2) reduces the
number of demonstrations required for fine-tuning, and (3) aligns the agent to
individual user task preferences.

本文提出一种交互式框架，利用用户反馈直接识别个性化任务无关概念生成反事实演示，用于数据增强并获得适合个性化用户目标的策略，通过在真实人类用户的离散和连续控制任务上进行实验证明了该方法有效降低了微调所需演示数量、使用户更好地了解机器人代理失败并使代理服务于个人任务偏好。

诊断、反馈、适应：人在执行时进行策略适应的框架

Diagnosis, Feedback, Adaptation: A Human-in-the-Loop Framework for  Test-Time Policy Adaptation

Large language models (LLMs) encode parametric knowledge about world facts
and have shown remarkable performance in knowledge-driven NLP tasks. However,
their reliance on parametric knowledge may cause them to overlook contextual
cues, leading to incorrect predictions in context-sensitive NLP tasks (e.g.,
knowledge acquisition tasks). In this paper, we seek to assess and enhance
LLMs' contextual faithfulness in two aspects: knowledge conflict and prediction
with abstention. We demonstrate that LLMs' faithfulness can be significantly
improved using carefully designed prompting strategies. In particular, we
identify opinion-based prompts and counterfactual demonstrations as the most
effective methods. Opinion-based prompts reframe the context as a narrator's
statement and inquire about the narrator's opinions, while counterfactual
demonstrations use instances containing false facts to improve faithfulness in
knowledge conflict situations. Neither technique requires additional training.
We conduct experiments on three datasets of two standard NLP tasks, machine
reading comprehension and relation extraction, and the results demonstrate
significant improvement in faithfulness to contexts.

本文探讨了大型语言模型在知识获取任务中通过设计启示策略，特别是意见为基础的提示和反事实演示，有效提高了上下文的真实性，并在三个数据集上进行实验，结果表明在上下文中的忠实度有了显著提高。