When robots learn reward functions using high capacity models that take raw
state directly as input, they need to both learn a representation for what
matters in the task -- the task ``features" -- as well as how to combine these
features into a single objective. If they try to do both at once from input
designed to teach the full reward function, it is easy to end up with a
representation that contains spurious correlations in the data, which fails to
generalize to new settings. Instead, our ultimate goal is to enable robots to
identify and isolate the causal features that people actually care about and
use when they represent states and behavior. Our idea is that we can tune into
this representation by asking users what behaviors they consider similar:
behaviors will be similar if the features that matter are similar, even if
low-level behavior is different; conversely, behaviors will be different if
even one of the features that matter differs. This, in turn, is what enables
the robot to disambiguate between what needs to go into the representation
versus what is spurious, as well as what aspects of behavior can be compressed
together versus not. The notion of learning representations based on similarity
has a nice parallel in contrastive learning, a self-supervised representation
learning technique that maps visually similar data points to similar
embeddings, where similarity is defined by a designer through data augmentation
heuristics. By contrast, in order to learn the representations that people use,
so we can learn their preferences and objectives, we use their definition of
similarity. In simulation as well as in a user study, we show that learning
through such similarity queries leads to representations that, while far from
perfect, are indeed more generalizable than self-supervised and task-input
alternatives.

从人们定义的相似度中学习表示使机器人能够识别和隔离人们实际关心和使用的因果特征，从而更好地学习他们的喜好和目标。在模拟和用户研究中，我们证明通过这种相似度查询学习可以比自我监督和任务输入等传统的表示学习方法得到更具有一般性的表示。

SIRL：基于相似性的隐式表示学习

SIRL: Similarity-based Implicit Representation Learning

We introduce an unsupervised approach to efficiently discover the underlying
features in a data set via crowdsourcing. Our queries ask crowd members to
articulate a feature common to two out of three displayed examples. In addition
we also ask the crowd to provide binary labels to the remaining examples based
on the discovered features. The triples are chosen adaptively based on the
labels of the previously discovered features on the data set. In two natural
models of features, hierarchical and independent, we show that a simple
adaptive algorithm, using "two-out-of-three" similarity queries, recovers all
features with less labor than any nonadaptive algorithm. Experimental results
validate the theoretical findings.

本文提出一种利用众包进行无监督学习的方法，通过 “二选一同类” 问题查询来发现数据特征，并使用自适应算法从三元组中恢复所有特征，实验证明其效率高于传统非自适应算法。