In the constant updates of the product dialogue systems, we need to retrain
the natural language understanding (NLU) model as new data from the real users
would be merged into the existent data accumulated in the last updates. Within
the newly added data, new intents would emerge and might have semantic
entanglement with the existing intents, e.g. new intents that are semantically
too specific or generic are actually subset or superset of some existing
intents in the semantic space, thus impairing the robustness of the NLU model.
As the first attempt to solve this problem, we setup a new benchmark consisting
of 4 Dialogue Version Control dataSets (DialogVCS). We formulate the intent
detection with imperfect data in the system update as a multi-label
classification task with positive but unlabeled intents, which asks the models
to recognize all the proper intents, including the ones with semantic
entanglement, in the inference. We also propose comprehensive baseline models
and conduct in-depth analyses for the benchmark, showing that the semantically
entangled intents can be effectively recognized with an automatic workflow.

通过 DialogVCS 建立了一种新的基准测试方法，将包含语义交织的意图识别问题建模为带有正标签但是未标注意图的多标签分类任务，并提出了综合的基线模型以有效地识别具有语义交错的意图。