The multi-scale information among the whole slide images (WSIs) is essential
for cancer diagnosis. Although the existing multi-scale vision Transformer has
shown its effectiveness for learning multi-scale image representation, it still
cannot work well on the gigapixel WSIs due to their extremely large image
sizes. To this end, we propose a novel Multi-scale Efficient Graph-Transformer
(MEGT) framework for WSI classification. The key idea of MEGT is to adopt two
independent Efficient Graph-based Transformer (EGT) branches to process the
low-resolution and high-resolution patch embeddings (i.e., tokens in a
Transformer) of WSIs, respectively, and then fuse these tokens via a
multi-scale feature fusion module (MFFM). Specifically, we design an EGT to
efficiently learn the local-global information of patch tokens, which
integrates the graph representation into Transformer to capture spatial-related
information of WSIs. Meanwhile, we propose a novel MFFM to alleviate the
semantic gap among different resolution patches during feature fusion, which
creates a non-patch token for each branch as an agent to exchange information
with another branch by cross-attention. In addition, to expedite network
training, a novel token pruning module is developed in EGT to reduce the
redundant tokens. Extensive experiments on TCGA-RCC and CAMELYON16 datasets
demonstrate the effectiveness of the proposed MEGT.

本文提出了一种名为 MEGT 的创新型多尺度高效图形变压器框架，用于 WSI 分类，并采用两个独立的 EGT 分支处理 WSI 中的低分辨率和高分辨率补丁嵌入以及通过多尺度特征融合模块融合这些令牌。