We present a framework for robots to learn novel visual concepts and tasks
via in-situ linguistic interactions with human users. Previous approaches have
either used large pre-trained visual models to infer novel objects zero-shot,
or added novel concepts along with their attributes and representations to a
concept hierarchy. We extend the approaches that focus on learning visual
concept hierarchies by enabling them to learn novel concepts and solve unseen
robotics tasks with them. To enable a visual concept learner to solve robotics
tasks one-shot, we developed two distinct techniques. Firstly, we propose a
novel approach, Hi-Viscont(HIerarchical VISual CONcept learner for Task), which
augments information of a novel concept to its parent nodes within a concept
hierarchy. This information propagation allows all concepts in a hierarchy to
update as novel concepts are taught in a continual learning setting. Secondly,
we represent a visual task as a scene graph with language annotations, allowing
us to create novel permutations of a demonstrated task zero-shot in-situ. We
present two sets of results. Firstly, we compare Hi-Viscont with the baseline
model (FALCON) on visual question answering(VQA) in three domains. While being
comparable to the baseline model on leaf level concepts, Hi-Viscont achieves an
improvement of over 9% on non-leaf concepts on average. We compare our model's
performance against the baseline FALCON model. Our framework achieves 33%
improvements in success rate metric, and 19% improvements in the object level
accuracy compared to the baseline model. With both of these results we
demonstrate the ability of our model to learn tasks and concepts in a continual
learning setting on the robot.

我们提出了一个框架，使机器人能够通过与人类用户进行当场语言交互来学习新的视觉概念和任务。通过在概念层次结构中给予新概念的父节点增加信息传播，我们提出了一种新方法 Hi-Viscont，该方法允许学习视觉概念层次结构的方法学习新的概念并解决未见过的机器人任务。我们还将视觉任务表示为带有语言注释的场景图，从而使我们能够零样例实时创建任务的新排列组合。通过与基线模型比较 Hi-Viscont 在视觉问答中的表现，我们得出一系列结果。在叶节点概念上，Hi-Viscont 与基线模型具备可比性，但在非叶节点概念上平均提高了超过 9%。对比基线模型，我们的框架在成功率、物体级准确率上分别提高了 33% 和 19%，从而证明了我们的模型在机器人的持续学习环境中学习任务和概念的能力。