Scene text removal (STR) aims at replacing text strokes in natural scenes
with visually coherent backgrounds. Recent STR approaches rely on iterative
refinements or explicit text masks, resulting in higher complexity and
sensitivity to the accuracy of text localization. Moreover, most existing STR
methods utilize convolutional neural networks (CNNs) for feature representation
while the potential of vision Transformers (ViTs) remains largely unexplored.
In this paper, we propose a simple-yet-effective ViT-based text eraser, dubbed
ViTEraser. Following a concise encoder-decoder framework, different types of
ViTs can be easily integrated into ViTEraser to enhance the long-range
dependencies and global reasoning. Specifically, the encoder hierarchically
maps the input image into the hidden space through ViT blocks and patch
embedding layers, while the decoder gradually upsamples the hidden features to
the text-erased image with ViT blocks and patch splitting layers. As ViTEraser
implicitly integrates text localization and inpainting, we propose a novel
end-to-end pretraining method, termed SegMIM, which focuses the encoder and
decoder on the text box segmentation and masked image modeling tasks,
respectively. To verify the effectiveness of the proposed methods, we
comprehensively explore the architecture, pretraining, and scalability of the
ViT-based encoder-decoder for STR, which provides deep insights into the
application of ViT to STR. Experimental results demonstrate that ViTEraser with
SegMIM achieves state-of-the-art performance on STR by a substantial margin.
Furthermore, the extended experiment on tampered scene text detection
demonstrates the generality of ViTEraser to other tasks. We believe this paper
can inspire more research on ViT-based STR approaches. Code will be available
at this https URL

本研究提出了一种基于 ViTs 的简单而有效的文本擦除器，在编码器 - 解码器框架内集成文本定位和修复，通过 SegMIM 方法进行端到端预训练，实现了在场景文本去除方面的最先进性能提升，表明了 ViTs 在此领域的广泛应用前景。

ViTEraser: 利用视觉 Transformer 和 SegMIM 预训练技术进行场景文本擦除

ViTEraser: Harnessing the Power of Vision Transformers for Scene Text  Removal with SegMIM Pretraining

Text removal algorithms have been proposed for uni-lingual scripts with
regular shapes and layouts. However, to the best of our knowledge, a generic
text removal method which is able to remove all or user-specified text regions
regardless of font, script, language or shape is not available. Developing such
a generic text eraser for real scenes is a challenging task, since it inherits
all the challenges of multi-lingual and curved text detection and inpainting.
To fill this gap, we propose a mask-based text removal network (MTRNet). MTRNet
is a conditional adversarial generative network (cGAN) with an auxiliary mask.
The introduced auxiliary mask not only makes the cGAN a generic text eraser,
but also enables stable training and early convergence on a challenging
large-scale synthetic dataset, initially proposed for text detection in real
scenes. What's more, MTRNet achieves state-of-the-art results on several
real-world datasets including ICDAR 2013, ICDAR 2017 MLT, and CTW1500, without
being explicitly trained on this data, outperforming previous state-of-the-art
methods trained directly on these datasets.

提出了一种基于掩模的文字删除方法（MTRNet），作为一种有条件的对抗性生成网络（cGAN），配备辅助掩模使其成为通用文本擦除工具，该算法在多个真实数据集上实现了最先进的结果。