Scene text recognition (STR) in the wild frequently encounters challenges
when coping with domain variations, font diversity, shape deformations, etc. A
straightforward solution is performing model fine-tuning tailored to a specific
scenario, but it is computationally intensive and requires multiple model
copies for various scenarios. Recent studies indicate that large language
models (LLMs) can learn from a few demonstration examples in a training-free
manner, termed "In-Context Learning" (ICL). Nevertheless, applying LLMs as a
text recognizer is unacceptably resource-consuming. Moreover, our pilot
experiments on LLMs show that ICL fails in STR, mainly attributed to the
insufficient incorporation of contextual information from diverse samples in
the training stage. To this end, we introduce E$^2$STR, a STR model trained
with context-rich scene text sequences, where the sequences are generated via
our proposed in-context training strategy. E$^2$STR demonstrates that a
regular-sized model is sufficient to achieve effective ICL capabilities in STR.
Extensive experiments show that E$^2$STR exhibits remarkable training-free
adaptation in various scenarios and outperforms even the fine-tuned
state-of-the-art approaches on public benchmarks.

Scene text recognition (STR) in the wild faces challenges due to domain variations, font diversity, shape deformations, etc. Recent studies show that large language models (LLMs) can learn from a few demonstration examples using In-Context Learning (ICL). However, applying LLMs as a text recognizer is resource-consuming. To address this, the paper introduces E$^2$STR, a STR model trained with context-rich scene text sequences, demonstrating effective ICL capabilities with a regular-sized model and outperforming fine-tuned approaches.

多模态上下文学习使自适应场景文本识别器

Multi-modal In-Context Learning Makes an Ego-evolving Scene Text  Recognizer

Contrastive Language-Image Pre-training (CLIP) models have shown promising
performance on zero-shot visual recognition tasks by learning visual
representations under natural language supervision. Recent studies attempt the
use of CLIP to tackle zero-shot anomaly detection by matching images with
normal and abnormal state prompts. However, since CLIP focuses on building
correspondence between paired text prompts and global image-level
representations, the lack of patch-level vision to text alignment limits its
capability on precise visual anomaly localization. In this work, we introduce a
training-free adaptation (TFA) framework of CLIP for zero-shot anomaly
localization. In the visual encoder, we innovate a training-free value-wise
attention mechanism to extract intrinsic local tokens of CLIP for patch-level
local description. From the perspective of text supervision, we particularly
design a unified domain-aware contrastive state prompting template. On top of
the proposed TFA, we further introduce a test-time adaptation (TTA) mechanism
to refine anomaly localization results, where a layer of trainable parameters
in the adapter is optimized using TFA's pseudo-labels and synthetic
noise-corrupted tokens. With both TFA and TTA adaptation, we significantly
exploit the potential of CLIP for zero-shot anomaly localization and
demonstrate the effectiveness of our proposed methods on various datasets.

使用 Contrastive Language-Image Pre-training (CLIP) 模型进行零样本异常检测，通过学习自然语言监督下的视觉表示，构建文本提示与全局图像级别表示之间的对应关系，采用训练自由的价值关注机制提取 CLIP 的内在局部标记以进行精确定位，设计一个统一的领域感知对比状态提示模板，通过测试时的自适应机制进一步优化异常定位结果。