Recent advance in deep learning has led to the rapid adoption of machine
learning-based NLP models in a wide range of applications. Despite the
continuous gain in accuracy, backward compatibility is also an important aspect
for industrial applications, yet it received little research attention.
Backward compatibility requires that the new model does not regress on cases
that were correctly handled by its predecessor. This work studies model update
regression in structured prediction tasks. We choose syntactic dependency
parsing and conversational semantic parsing as representative examples of
structured prediction tasks in NLP. First, we measure and analyze model update
regression in different model update settings. Next, we explore and benchmark
existing techniques for reducing model update regression including model
ensemble and knowledge distillation. We further propose a simple and effective
method, Backward-Congruent Re-ranking (BCR), by taking into account the
characteristics of structured prediction. Experiments show that BCR can better
mitigate model update regression than model ensemble and knowledge distillation
approaches.

本文研究了结构预测任务中的模型更新回归问题，通过测量和分析了不同模型更新设置下的模型更新回归情况，探索和基准测试了现有的降低模型更新回归的技术，包括模型集成和知识蒸馏，并提出了一种简单有效的方法，称为 “Backward-Congruent Re-ranking (BCR)”，该方法结合了结构预测的特性，比模型集成和知识蒸馏方法更好地缓解了模型更新回归问题。