Deep learning often faces the challenge of efficiently processing dynamic
inputs, such as sensor data or user inputs. For example, an AI writing
assistant is required to update its suggestions in real time as a document is
edited. Re-running the model each time is expensive, even with compression
techniques like knowledge distillation, pruning, or quantization. Instead, we
take an incremental computing approach, looking to reuse calculations as the
inputs change. However, the dense connectivity of conventional architectures
poses a major obstacle to incremental computation, as even minor input changes
cascade through the network and restrict information reuse. To address this, we
use vector quantization to discretize intermediate values in the network, which
filters out noisy and unnecessary modifications to hidden neurons, facilitating
the reuse of their values. We apply this approach to the transformers
architecture, creating an efficient incremental inference algorithm with
complexity proportional to the fraction of the modified inputs. Our experiments
with adapting the OPT-125M pre-trained language model demonstrate comparable
accuracy on document classification while requiring 12.1X (median) fewer
operations for processing sequences of atomic edits.

深度学习和增量计算方法，采用向量量化处理输入数据，通过离散化中间值以便复用信息，应用于 Transformer 架构，实现了复杂度与输入变动比例成正比的高效增量推理算法，并通过实验证明在文档分类中达到了和传统方法几乎相等的准确性，但处理微小编辑序列时所需操作次数降低到了原先的 12.1 倍。