Architecture search is the process of automatically learning the neural model
or cell structure that best suits the given task. Recently, this approach has
shown promising performance improvements (on language modeling and image
classification) with reasonable training speed, using a weight sharing strategy
called Efficient Neural Architecture Search (ENAS). In our work, we first
introduce a novel continual architecture search (CAS) approach, so as to
continually evolve the model parameters during the sequential training of
several tasks, without losing performance on previously learned tasks (via
block-sparsity and orthogonality constraints), thus enabling life-long
learning. Next, we explore a multi-task architecture search (MAS) approach over
ENAS for finding a unified, single cell structure that performs well across
multiple tasks (via joint controller rewards), and hence allows more
generalizable transfer of the cell structure knowledge to an unseen new task.
We empirically show the effectiveness of our sequential continual learning and
parallel multi-task learning based architecture search approaches on diverse
sentence-pair classification tasks (GLUE) and multimodal-generation based video
captioning tasks. Further, we present several ablations and analyses on the
learned cell structures.

本研究提出了一种新颖的连续体系结构搜索方法和一种基于多任务学习的组合单元结构搜索方法，通过块稀疏性和正交性约束实现连续的进化学习，同时通过联合控制器奖励实现学习到的单元结构在多个任务上的优秀表现，成功应用于句子对分类和视频字幕生成等多领域任务中。