Detecting successful behaviour is crucial for training intelligent agents. As
such, generalisable reward models are a prerequisite for agents that can learn
to generalise their behaviour. In this work we focus on developing robust
success detectors that leverage large, pretrained vision-language models
(Flamingo, Alayrac et al. (2022)) and human reward annotations. Concretely, we
treat success detection as a visual question answering (VQA) problem, denoted
SuccessVQA. We study success detection across three vastly different domains:
(i) interactive language-conditioned agents in a simulated household, (ii) real
world robotic manipulation, and (iii) "in-the-wild" human egocentric videos. We
investigate the generalisation properties of a Flamingo-based success detection
model across unseen language and visual changes in the first two domains, and
find that the proposed method is able to outperform bespoke reward models in
out-of-distribution test scenarios with either variation. In the last domain of
"in-the-wild" human videos, we show that success detection on unseen real
videos presents an even more challenging generalisation task warranting future
work. We hope our initial results encourage further work in real world success
detection and reward modelling.

本文介绍了一种使用大型预训练视觉语言模型和人工奖励注释的成功检测方法，将其作为视觉问答问题，研究了在三个不同领域的成功检测和推广属性，证明该方法能在看到未知语言和视觉变化的情况下胜过定制奖励模型。