Referring Expression Comprehension (REC) aims to localize an image region of
a given object described by a natural-language expression. While promising
performance has been demonstrated, existing REC algorithms make a strong
assumption that training data feeding into a model are given upfront, which
degrades its practicality for real-world scenarios. In this paper, we propose
Continual Referring Expression Comprehension (CREC), a new setting for REC,
where a model is learning on a stream of incoming tasks. In order to
continuously improve the model on sequential tasks without forgetting prior
learned knowledge and without repeatedly re-training from a scratch, we propose
an effective baseline method named Dual Modular Memorization (DMM), which
alleviates the problem of catastrophic forgetting by two memorization modules:
Implicit-Memory and Explicit-Memory. Specifically, the former module aims to
constrain drastic changes to important parameters learned on old tasks when
learning a new task; while the latter module maintains a buffer pool to
dynamically select and store representative samples of each seen task for
future rehearsal. We create three benchmarks for the new CREC setting, by
respectively re-splitting three widely-used REC datasets RefCOCO, RefCOCO+ and
RefCOCOg into sequential tasks. Extensive experiments on the constructed
benchmarks demonstrate that our DMM method significantly outperforms other
alternatives, based on two popular REC backbones. We make the source code and
benchmarks publicly available to foster future progress in this field:
this https URL

本文提出了 Continual Referring Expression Comprehension (CREC) 新的设置，通过 Dual Modular Memorization (DMM) 方法，不断改进模型在连续任务上的性能，避免遗忘之前学到的知识并减少重复重新训练的需求。作者在三个常用 REC 数据集上构建了新的 CREC 基准，并通过大量实验证明 DMM 方法显著优于其他方法。