Modular neural networks without additional training have recently been shown
to surpass end-to-end neural networks on challenging vision-language tasks. The
latest such methods simultaneously introduce LLM-based code generation to build
programs and a number of skill-specific, task-oriented modules to execute them.
In this paper, we focus on ViperGPT and ask where its additional performance
comes from and how much is due to the (state-of-art, end-to-end) BLIP-2 model
it subsumes vs. additional symbolic components. To do so, we conduct a
controlled study (comparing end-to-end, modular, and prompting-based methods
across several VQA benchmarks). We find that ViperGPT's reported gains over
BLIP-2 can be attributed to its selection of task-specific modules, and when we
run ViperGPT using a more task-agnostic selection of modules, these gains go
away. Additionally, ViperGPT retains much of its performance if we make
prominent alterations to its selection of modules: e.g. removing or retaining
only BLIP-2. Finally, we compare ViperGPT against a prompting-based
decomposition strategy and find that, on some benchmarks, modular approaches
significantly benefit by representing subtasks with natural language, instead
of code.

“ViperGPT” 对比 “BLIP-2 model” 发现，其额外性能主要来源于对具体任务的模块选择，而将其选用更兼容的模块时，其性能并不突出；此外，在某些基准测试中，模块化方法与自然语言表示子任务相比，收益显著。

分析视觉问题分解的模块化方法

Analyzing Modular Approaches for Visual Question Decomposition

Continual learning (CL) aims to train deep neural networks efficiently on
streaming data while limiting the forgetting caused by new tasks. However,
learning transferable knowledge with less interference between tasks is
difficult, and real-world deployment of CL models is limited by their inability
to measure predictive uncertainties. To address these issues, we propose
handling CL tasks with neural processes (NPs), a class of meta-learners that
encode different tasks into probabilistic distributions over functions all
while providing reliable uncertainty estimates. Specifically, we propose an
NP-based CL approach (NPCL) with task-specific modules arranged in a
hierarchical latent variable model. We tailor regularizers on the learned
latent distributions to alleviate forgetting. The uncertainty estimation
capabilities of the NPCL can also be used to handle the task head/module
inference challenge in CL. Our experiments show that the NPCL outperforms
previous CL approaches. We validate the effectiveness of uncertainty estimation
in the NPCL for identifying novel data and evaluating instance-level model
confidence. Code is available at https://github.com/srvCodes/NPCL.

使用神经过程处理连续学习任务，通过任务特定模块和层次潜在变量模型，提供可靠的不确定性估计，并在不同任务间减轻遗忘现象。

NPCL：针对不确定性感知的连续学习的神经过程

NPCL: Neural Processes for Uncertainty-Aware Continual Learning

Distribution shift (e.g., task or domain shift) in continual learning (CL)
usually results in catastrophic forgetting of neural networks. Although it can
be alleviated by repeatedly replaying buffered data, the every-step replay is
time-consuming. In this paper, we study which modules in neural networks are
more prone to forgetting by investigating their training dynamics during CL.
Our proposed metrics show that only a few modules are more task-specific and
sensitively alter between tasks, while others can be shared across tasks as
common knowledge. Hence, we attribute forgetting mainly to the former and find
that finetuning them only on a small buffer at the end of any CL method can
bring non-trivial improvement. Due to the small number of finetuned parameters,
such ``Forgetting Prioritized Finetuning (FPF)'' is efficient in computation.
We further propose a more efficient and simpler method that entirely removes
the every-step replay and replaces them by only $k$-times of FPF periodically
triggered during CL. Surprisingly, this ``$k$-FPF'' performs comparably to FPF
and outperforms the SOTA CL methods but significantly reduces their
computational overhead and cost. In experiments on several benchmarks of class-
and domain-incremental CL, FPF consistently improves existing CL methods by a
large margin, and $k$-FPF further excels in efficiency without degrading the
accuracy. We also empirically studied the impact of buffer size, epochs per
task, and finetuning modules on the cost and accuracy of our methods.

本文针对持续学习过程中的分布漂移（如任务或领域漂移）导致神经网络对原有任务的忘记问题，研究了神经网络中哪些模块更容易被遗忘，提出了遗忘优先微调（FPF）和基于周期触发的 $k$-FPF 两种方法，取得了较好的效果并显著降低了计算成本。实验表明，FPF 方法在几个增量 CL 基准测试中均能显著提高方法效果，$k$-FPF 进一步提高了效率而不降低准确性。