This work addresses the need for a balanced approach between performance and
efficiency in scalable production environments for visually-rich document
understanding (VDU) tasks. Currently, there is a reliance on large document
foundation models that offer advanced capabilities but come with a heavy
computational burden. In this paper, we propose a multimodal early exit (EE)
model design that incorporates various training strategies, exit layer types
and placements. Our goal is to achieve a Pareto-optimal balance between
predictive performance and efficiency for multimodal document image
classification. Through a comprehensive set of experiments, we compare our
approach with traditional exit policies and showcase an improved
performance-efficiency trade-off. Our multimodal EE design preserves the
model's predictive capabilities, enhancing both speed and latency. This is
achieved through a reduction of over 20% in latency, while fully retaining the
baseline accuracy. This research represents the first exploration of multimodal
EE design within the VDU community, highlighting as well the effectiveness of
calibration in improving confidence scores for exiting at different layers.
Overall, our findings contribute to practical VDU applications by enhancing
both performance and efficiency.

本研究提出了一种多模态早期退出模型设计，旨在在可视化丰富的文档理解（VDU）任务的可扩展生产环境中实现预测性能和效率之间的 Pareto 最优平衡。通过一系列实验证明，我们的方法与传统的退出策略相比，在改善了性能效率权衡方面取得了显著的改进。该研究对于提高 VDU 应用的性能和效率具有实际意义。

多模态自适应推理与任意时刻提前退出的文档图像分类

Multimodal Adaptive Inference for Document Image Classification with  Anytime Early Exiting

Typical vision backbones manipulate structured features. As a compromise,
semantic segmentation has long been modeled as per-point prediction on dense
regular grids. In this work, we present a novel and efficient modeling that
starts from interpreting the image as a tessellation of learnable regions, each
of which has flexible geometrics and carries homogeneous semantics. To model
region-wise context, we exploit Transformer to encode regions in a
sequence-to-sequence manner by applying multi-layer self-attention on the
region embeddings, which serve as proxies of specific regions. Semantic
segmentation is now carried out as per-region prediction on top of the encoded
region embeddings using a single linear classifier, where a decoder is no
longer needed. The proposed RegProxy model discards the common Cartesian
feature layout and operates purely at region level. Hence, it exhibits the most
competitive performance-efficiency trade-off compared with the conventional
dense prediction methods. For example, on ADE20K, the small-sized RegProxy-S/16
outperforms the best CNN model using 25% parameters and 4% computation, while
the largest RegProxy-L/16 achieves 52.9mIoU which outperforms the
state-of-the-art by 2.1% with fewer resources. Codes and models are available
at this https URL

本文提出了一种名为 RegProxy 的模型，通过 Transformer 实现了以区域为中心的上下文编码和预测，相比现有的方法在性能和效率上取得了更好的平衡。