Humans acquire language continually with much more limited access to data
samples at a time, as compared to contemporary NLP systems. To study this
human-like language acquisition ability, we present VisCOLL, a visually
grounded language learning task, which simulates the continual acquisition of
compositional phrases from streaming visual scenes. In the task, models are
trained on a paired image-caption stream which has shifting object
distribution; while being constantly evaluated by a visually-grounded masked
language prediction task on held-out test sets. VisCOLL compounds the
challenges of continual learning (i.e., learning from continuously shifting
data distribution) and compositional generalization (i.e., generalizing to
novel compositions). To facilitate research on VisCOLL, we construct two
datasets, COCO-shift and Flickr-shift, and benchmark them using different
continual learning methods. Results reveal that SoTA continual learning
approaches provide little to no improvements on VisCOLL, since storing examples
of all possible compositions is infeasible. We conduct further ablations and
analysis to guide future work.

本文介绍了 VisCOLL, 一种模拟从流动视觉场景中不断习得组成短语的视觉感知语言学习任务，其使得机器模型习得人类一样的自然语言习得能力，本文构建了两个数据集，用不同的连续学习方法来比较并测试了它们。