This technical report presents LongViT, a vision Transformer that can process
gigapixel images in an end-to-end manner. Specifically, we split the gigapixel
image into a sequence of millions of patches and project them linearly into
embeddings. LongNet is then employed to model the extremely long sequence,
generating representations that capture both short-range and long-range
dependencies. The linear computation complexity of LongNet, along with its
distributed algorithm, enables us to overcome the constraints of both
computation and memory. We apply LongViT in the field of computational
pathology, aiming for cancer diagnosis and prognosis within gigapixel
whole-slide images. Experimental results demonstrate that LongViT effectively
encodes gigapixel images and outperforms previous state-of-the-art methods on
cancer subtyping and survival prediction. Code and models will be available at
this https URL

本技术报告介绍了 LongViT，一种能够以端到端方式处理十亿像素图像的视觉 Transformer。我们将十亿像素图像分割成数百万个补丁，并线性投影到嵌入中。然后，我们使用 LongNet 对这个极长序列进行建模，生成捕捉了短程和长程依赖关系的表示。LongNet 的线性计算复杂度和分布式算法克服了计算和内存的限制。我们将 LongViT 应用于计算病理学领域，旨在进行癌症诊断和预后的十亿像素全切片图像。实验结果表明，LongViT 能够有效编码十亿像素图像，并在癌症亚型分型和生存预测方面优于先前的最先进方法。代码和模型将在此 https URL 上提供。