Modularity is a compelling solution to continual learning (CL), the problem
of modeling sequences of related tasks. Learning and then composing modules to
solve different tasks provides an abstraction to address the principal
challenges of CL including catastrophic forgetting, backward and forward
transfer across tasks, and sub-linear model growth. We introduce local module
composition (LMC), an approach to modular CL where each module is provided a
local structural component that estimates a module's relevance to the input.
Dynamic module composition is performed layer-wise based on local relevance
scores. We demonstrate that agnosticity to task identities (IDs) arises from
(local) structural learning that is module-specific as opposed to the task-
and/or model-specific as in previous works, making LMC applicable to more CL
settings compared to previous works. In addition, LMC also tracks statistics
about the input distribution and adds new modules when outlier samples are
detected. In the first set of experiments, LMC performs favorably compared to
existing methods on the recent Continual Transfer-learning Benchmark without
requiring task identities. In another study, we show that the locality of
structural learning allows LMC to interpolate to related but unseen tasks
(OOD), as well as to compose modular networks trained independently on
different task sequences into a third modular network without any fine-tuning.
Finally, in search for limitations of LMC we study it on more challenging
sequences of 30 and 100 tasks, demonstrating that local module selection
becomes much more challenging in presence of a large number of candidate
modules. In this setting best performing LMC spawns much fewer modules compared
to an oracle based baseline, however, it reaches a lower overall accuracy. The
codebase is available under this https URL

通过探索局部模块组合（LMC）的方法，使用结构性的学习和模块选择，LMC 可以为连续学习提供较好的解决方案。