Internal Language Model (LM)-based methods use permutation language modeling
(PLM) to solve the error correction caused by conditional independence in
external LM-based methods. However, random permutations of human interference
cause fit oscillations in the model training, and Iterative Refinement (IR)
operation to improve multimodal information decoupling also introduces
additional overhead. To address these issues, this paper proposes the
Hierarchical Attention autoregressive Model with Adaptive Permutation (HAAP) to
enhance the location-context-image interaction capability, improving
autoregressive generalization with internal LM. First, we propose Implicit
Permutation Neurons (IPN) to generate adaptive attention masks to dynamically
exploit token dependencies. The adaptive masks increase the diversity of
training data and prevent model dependency on a specific order. It reduces the
training overhead of PLM while avoiding training fit oscillations. Second, we
develop Cross-modal Hierarchical Attention mechanism (CHA) to couple context
and image features. This processing establishes rich positional semantic
dependencies between context and image while avoiding IR. Extensive
experimental results show the proposed HAAP achieves state-of-the-art (SOTA)
performance in terms of accuracy, complexity, and latency on several datasets.

本文提出了一种使用自适应置换的分级注意力自回归模型（HAAP）来增强位置 - 上下文 - 图像交互能力，从而改进了内部语言模型的自回归泛化。通过使用隐式置换神经元（IPN）生成自适应的注意力掩码，增加了训练数据的多样性，避免了模型对特定顺序的依赖，并减少了置换语言模型的训练开销。同时，通过交叉模态分级注意机制（CHA）将上下文和图像特征相结合，建立了丰富的位置语义依赖关系，避免了迭代细化操作。大量实验结果表明，所提出的 HAAP 在准确性、复杂性和延迟方面具有最先进的性能。

HAAP: 基于视觉 - 上下文分层注意力自回归与自适应排列的场景文本识别

HAAP: Vision-context Hierarchical Attention Autoregressive with Adaptive  Permutation for Scene Text Recognition

Context-aware STR methods typically use internal autoregressive (AR) language
models (LM). Inherent limitations of AR models motivated two-stage methods
which employ an external LM. The conditional independence of the external LM on
the input image may cause it to erroneously rectify correct predictions,
leading to significant inefficiencies. Our method, PARSeq, learns an ensemble
of internal AR LMs with shared weights using Permutation Language Modeling. It
unifies context-free non-AR and context-aware AR inference, and iterative
refinement using bidirectional context. Using synthetic training data, PARSeq
achieves state-of-the-art (SOTA) results in STR benchmarks (91.9% accuracy) and
more challenging datasets. It establishes new SOTA results (96.0% accuracy)
when trained on real data. PARSeq is optimal on accuracy vs parameter count,
FLOPS, and latency because of its simple, unified structure and parallel token
processing. Due to its extensive use of attention, it is robust on
arbitrarily-oriented text which is common in real-world images. Code,
pretrained weights, and data are available at: this https URL.

PARSeq 使用 Permutation Language Modeling 学习了一组共享权重的内部 AR 语言模型的集合，实现了无上下文的非 AR 和有上下文的 AR 推理以及使用双向上下文的迭代细化，使用注意力机制进行文本识别，取得了在 STR 基准测试中的最新成果，并在一些具有挑战性的数据集上建立了新的 SOTA 结果。