Measuring the confidence of AI models is critical for safely deploying AI in real-world industrial systems. One important application of confidence measurement is information extraction from scanned documents. However, there exists no solution to provide reliable confidence score for current state-of-the-art deep-learning-based information extractors. In this paper, we propose a complete and novel architecture to measure confidence of current deep learning models in document information extraction task. Our architecture consists of a Multi-modal Conformal Predictor and a Variational Cluster-oriented Anomaly Detector, trained to faithfully estimate its confidence on its outputs without the need of host models modification. We evaluate our architecture on real-wold datasets, not only outperforming competing confidence estimators by a huge margin but also demonstrating generalization ability to out-of-distribution data.

本文提出了一种用于测量当前深度学习模型在文档信息提取任务中置信度的完整新型体系结构，该体系结构包括一种多模态符合预测器和一种变分聚类定向异常检测器，并在真实世界数据集上进行了评估，不仅大大优于竞争对手的置信度估计器，而且还展示了对超出分布数据的泛化能力。

HYCEDIS: 深度文档智能系统的混合置信引擎