Although deep learning has enabled a huge leap forward in image inpainting,
current methods are often unable to synthesize realistic high-frequency
details. In this paper, we propose applying super-resolution to coarsely
reconstructed outputs, refining them at high resolution, and then downscaling
the output to the original resolution. By introducing high-resolution images to
the refinement network, our framework is able to reconstruct finer details that
are usually smoothed out due to spectral bias - the tendency of neural networks
to reconstruct low frequencies better than high frequencies. To assist training
the refinement network on large upscaled holes, we propose a progressive
learning technique in which the size of the missing regions increases as
training progresses. Our zoom-in, refine and zoom-out strategy, combined with
high-resolution supervision and progressive learning, constitutes a
framework-agnostic approach for enhancing high-frequency details that can be
applied to any CNN-based inpainting method. We provide qualitative and
quantitative evaluations along with an ablation analysis to show the
effectiveness of our approach. This seemingly simple, yet powerful approach,
outperforms state-of-the-art inpainting methods. Our code is available in
this https URL

本文提出一种将超分辨率应用于粗略重建输出并在高分辨率下进行细化，然后将输出缩小到原始分辨率的框架，以提高高频细节的重建效果，同时采用渐进式学习技术来辅助大空缺区域的训练，获得了比现有最佳方法更好的表现。

Zoom-to-Inpaint：具备高频细节的图像修复算法

Zoom-to-Inpaint: Image Inpainting with High-Frequency Details

Over-parameterized deep neural networks (DNNs) with sufficient capacity to
memorize random noise can achieve excellent generalization performance,
challenging the bias-variance trade-off in classical learning theory. Recent
studies claimed that DNNs first learn simple patterns and then memorize noise;
some other works showed a phenomenon that DNNs have a spectral bias to learn
target functions from low to high frequencies during training. However, we show
that the monotonicity of the learning bias does not always hold: under the
experimental setup of deep double descent, the high-frequency components of
DNNs diminish in the late stage of training, leading to the second descent of
the test error. Besides, we find that the spectrum of DNNs can be applied to
indicating the second descent of the test error, even though it is calculated
from the training set only.

研究表明，深度神经网络的学习偏差会在训练的后期阶段出现第二次下降，其高频分量也会因此减少，导致测试误差的第二次下降，而网络的频谱可以用于指示测试误差的第二次下降。

重新思考 DNN 的泛化、记忆和谱偏差之间的关系

Rethink the Connections among Generalization, Memorization and the  Spectral Bias of DNNs

Deep neural networks have achieved remarkable success in computer vision
tasks. Existing neural networks mainly operate in the spatial domain with fixed
input sizes. For practical applications, images are usually large and have to
be downsampled to the predetermined input size of neural networks. Even though
the downsampling operations reduce computation and the required communication
bandwidth, it removes both redundant and salient information obliviously, which
results in accuracy degradation. Inspired by digital signal processing
theories, we analyze the spectral bias from the frequency perspective and
propose a learning-based frequency selection method to identify the trivial
frequency components which can be removed without accuracy loss. The proposed
method of learning in the frequency domain leverages identical structures of
the well-known neural networks, such as ResNet-50, MobileNetV2, and Mask R-CNN,
while accepting the frequency-domain information as the input. Experiment
results show that learning in the frequency domain with static channel
selection can achieve higher accuracy than the conventional spatial
downsampling approach and meanwhile further reduce the input data size.
Specifically for ImageNet classification with the same input size, the proposed
method achieves 1.41% and 0.66% top-1 accuracy improvements on ResNet-50 and
MobileNetV2, respectively. Even with half input size, the proposed method still
improves the top-1 accuracy on ResNet-50 by 1%. In addition, we observe a 0.8%
average precision improvement on Mask R-CNN for instance segmentation on the
COCO dataset.

通过在频域进行学习和频道选择，我们提出了减少冗余和显著信息以提高图像分类精度的学习方法，得出使用该方法的 ResNet-50、MobileNetV2 和 Mask R-CNN 在图像分类和实例分割上表现更好的结论。

频域学习

Learning in the Frequency Domain

An intriguing phenomenon observed during training neural networks is the
spectral bias, which states that neural networks are biased towards learning
less complex functions. The priority of learning functions with low complexity
might be at the core of explaining generalization ability of neural network,
and certain efforts have been made to provide theoretical explanation for
spectral bias. However, there is still no satisfying theoretical result
justifying the underlying mechanism of spectral bias. In this paper, we give a
comprehensive and rigorous explanation for spectral bias and relate it with the
neural tangent kernel function proposed in recent work. We prove that the
training process of neural networks can be decomposed along different
directions defined by the eigenfunctions of the neural tangent kernel, where
each direction has its own convergence rate and the rate is determined by the
corresponding eigenvalue. We then provide a case study when the input data is
uniformly distributed over the unit sphere, and show that lower degree
spherical harmonics are easier to be learned by over-parameterized neural
networks. Finally, we provide numerical experiments to demonstrate the
correctness of our theory. Our experimental results also show that our theory
can tolerate certain model misspecification in terms of the input data
distribution.

论文通过与神经切向核函数的关联性进行解释，给出了关于神经网络中谱偏置的全面而严格的解释，并证明了神经网络的训练过程可以沿神经切向核的不同方向分解，每个方向都有自己的收敛率，而该收敛率由相应的特征值确定。