Tangent Model Composition (TMC) is a method to combine component models
independently fine-tuned around a pre-trained point. Component models are
tangent vectors to the pre-trained model that can be added, scaled, or
subtracted to support incremental learning, ensembling, or unlearning.
Component models are composed at inference time via scalar combination,
reducing the cost of ensembling to that of a single model. TMC improves
accuracy by 4.2% compared to ensembling non-linearly fine-tuned models at a
2.5x to 10x reduction of inference cost, growing linearly with the number of
component models. Each component model can be forgotten at zero cost, with no
residual effect on the resulting inference. When used for continual
fine-tuning, TMC is not constrained by sequential bias and can be executed in
parallel on federated data. TMC outperforms recently published continual
fine-tuning methods almost uniformly on each setting -- task-incremental,
class-incremental, and data-incremental -- on a total of 13 experiments across
3 benchmark datasets, despite not using any replay buffer. TMC is designed for
composing models that are local to a pre-trained embedding, but could be
extended to more general settings.

Tangent Model Composition (TMC) 是一种将组件模型独立微调到预训练点周围并组合的方法，旨在支持增量学习、组合或取消学习，并在推理时通过标量组合组合组件模型，从而将集成成本降低到单个模型的成本以提高准确度，在 13 个实验和 3 个基准数据集上相对于非线性微调模型的组合在推理成本减少 2.5 倍到 10 倍的同时提高了 4.2% 的准确度，适用于增量微调、并行处理，无须重播缓冲区。