Transformer based methods have achieved great success in image inpainting
recently. However, we find that these solutions regard each pixel as a token,
thus suffering from an information loss issue from two aspects: 1) They
downsample the input image into much lower resolutions for efficiency
consideration. 2) They quantize $256^3$ RGB values to a small number (such as
512) of quantized color values. The indices of quantized pixels are used as
tokens for the inputs and prediction targets of the transformer. To mitigate
these issues, we propose a new transformer based framework called "PUT".
Specifically, to avoid input downsampling while maintaining computation
efficiency, we design a patch-based auto-encoder P-VQVAE. The encoder converts
the masked image into non-overlapped patch tokens and the decoder recovers the
masked regions from the inpainted tokens while keeping the unmasked regions
unchanged. To eliminate the information loss caused by input quantization, an
Un-quantized Transformer is applied. It directly takes features from the
P-VQVAE encoder as input without any quantization and only regards the
quantized tokens as prediction targets. Furthermore, to make the inpainting
process more controllable, we introduce semantic and structural conditions as
extra guidance. Extensive experiments show that our method greatly outperforms
existing transformer based methods on image fidelity and achieves much higher
diversity and better fidelity than state-of-the-art pluralistic inpainting
methods on complex large-scale datasets (e.g., ImageNet). Codes are available
at this https URL

基于 Transformer 的图像修复方法存在信息丢失问题，为解决这个问题，我们提出了 PUT 框架，利用自编码器和非量化 Transformer 来保留图像信息并增加多样性。