Measuring the confidence of AI models is critical for safely deploying AI in
real-world industrial systems. One important application of confidence
measurement is information extraction from scanned documents. However, there
exists no solution to provide reliable confidence score for current
state-of-the-art deep-learning-based information extractors. In this paper, we
propose a complete and novel architecture to measure confidence of current deep
learning models in document information extraction task. Our architecture
consists of a Multi-modal Conformal Predictor and a Variational
Cluster-oriented Anomaly Detector, trained to faithfully estimate its
confidence on its outputs without the need of host models modification. We
evaluate our architecture on real-wold datasets, not only outperforming
competing confidence estimators by a huge margin but also demonstrating
generalization ability to out-of-distribution data.

本文提出了一种用于测量当前深度学习模型在文档信息提取任务中置信度的完整新型体系结构，该体系结构包括一种多模态符合预测器和一种变分聚类定向异常检测器，并在真实世界数据集上进行了评估，不仅大大优于竞争对手的置信度估计器，而且还展示了对超出分布数据的泛化能力。

HYCEDIS: 深度文档智能系统的混合置信引擎

HYCEDIS: HYbrid Confidence Engine for Deep Document Intelligence System

The reliable measurement of confidence in classifiers' predictions is very
important for many applications and is, therefore, an important part of
classifier design. Yet, although deep learning has received tremendous
attention in recent years, not much progress has been made in quantifying the
prediction confidence of neural network classifiers. Bayesian models offer a
mathematically grounded framework to reason about model uncertainty, but
usually come with prohibitive computational costs. In this paper we propose a
simple, scalable method to achieve a reliable confidence score, based on the
data embedding derived from the penultimate layer of the network. We
investigate two ways to achieve desirable embeddings, by using either a
distance-based loss or Adversarial Training. We then test the benefits of our
method when used for classification error prediction, weighting an ensemble of
classifiers, and novelty detection. In all tasks we show significant
improvement over traditional, commonly used confidence scores.

本文提出了一种基于神经网络中数据嵌入的简单可扩展的可靠置信度评分方法，通过距离损失或对抗训练对得到的嵌入进行处理，比传统置信度评分在分类错误预测、加权分类器集成以及新颖性检测中表现更加优异。