High-resolution images are prevalent in various applications, such as
autonomous driving and computer-aided diagnosis. However, training neural
networks on such images is computationally challenging and easily leads to
out-of-memory errors even on modern GPUs. We propose a simple method, Iterative
Patch Selection (IPS), which decouples the memory usage from the input size and
thus enables the processing of arbitrarily large images under tight hardware
constraints. IPS achieves this by selecting only the most salient patches,
which are then aggregated into a global representation for image recognition.
For both patch selection and aggregation, a cross-attention based transformer
is introduced, which exhibits a close connection to Multiple Instance Learning.
Our method demonstrates strong performance and has wide applicability across
different domains, training regimes and image sizes while using minimal
accelerator memory. For example, we are able to finetune our model on
whole-slide images consisting of up to 250k patches (>16 gigapixels) with only
5 GB of GPU VRAM at a batch size of 16.

提出一种名为 Iterative Patch Selection (IPS) 的简单方法，该方法仅选择最显着的补丁，将其聚合成用于图像识别的全局表示，从而可以在硬件限制下处理任意大小的图像，展现出强大的性能和广泛的适用性，同时使用最少的加速器内存。