We propose a dual-stream multi-scale vision transformer (DS-MSHViT)
architecture that processes RGB and optical flow inputs for efficient sewer
defect classification. Unlike existing methods that combine the predictions of
two separate networks trained on each modality, we jointly train a single
network with two branches for RGB and motion. Our key idea is to use
self-attention regularization to harness the complementary strengths of the RGB
and motion streams. The motion stream alone struggles to generate accurate
attention maps, as motion images lack the rich visual features present in RGB
images. To facilitate this, we introduce an attention consistency loss between
the dual streams. By leveraging motion cues through a self-attention
regularizer, we align and enhance RGB attention maps, enabling the network to
concentrate on pertinent input regions. We evaluate our data on a public
dataset as well as cross-validate our model performance in a novel dataset. Our
method outperforms existing models that utilize either convolutional neural
networks (CNNs) or multi-scale hybrid vision transformers (MSHViTs) without
employing attention regularization between the two streams.

我们提出了一种双流多尺度视觉转换器 (DS-MSHViT) 架构，用于高效下水道缺陷分类，并联训练了一个 RGB 分支和一个运动分支的单一网络，通过自注意力正则化来利用 RGB 和运动流的互补优势，以增强 RGB 的关注地图从而集中于相关输入区域。

双流注意力变换器用于下水道缺陷分类

Dual-Stream Attention Transformers for Sewer Defect Classification

Sparse Transformer has recently attracted a lot of attention since the
ability for reducing the quadratic dependency on the sequence length. We argue
that two factors, information bottleneck sensitivity and inconsistency between
different attention topologies, could affect the performance of the Sparse
Transformer. This paper proposes a well-designed model named ERNIE-Sparse. It
consists of two distinctive parts: (i) Hierarchical Sparse Transformer (HST) to
sequentially unify local and global information. (ii) Self-Attention
Regularization (SAR) method, a novel regularization designed to minimize the
distance for transformers with different attention topologies. To evaluate the
effectiveness of ERNIE-Sparse, we perform extensive evaluations. Firstly, we
perform experiments on a multi-modal long sequence modeling task benchmark,
Long Range Arena (LRA). Experimental results demonstrate that ERNIE-Sparse
significantly outperforms a variety of strong baseline methods including the
dense attention and other efficient sparse attention methods and achieves
improvements by 2.77% (57.78% vs. 55.01%). Secondly, to further show the
effectiveness of our method, we pretrain ERNIE-Sparse and verified it on 3 text
classification and 2 QA downstream tasks, achieve improvements on
classification benchmark by 0.83% (92.46% vs. 91.63%), on QA benchmark by 3.24%
(74.67% vs. 71.43%). Experimental results continue to demonstrate its superior
performance.

提出 ERNIE-Sparse 模型，它包含两个不同的部分：分层稀疏 Transformer（Hierarchical Sparse Transformer，HST）和自注意力正则化（Self-Attention Regularization，SAR）方法。实验证明，在长序列建模任务和文本分类任务中，ERNIE-Sparse 显著优于其他基线方法。