Recently, end-to-end transformer-based detectors (DETRs) have achieved
remarkable performance. However, the issue of the high computational cost of
DETRs has not been effectively addressed, limiting their practical application
and preventing them from fully exploiting the benefits of no post-processing,
such as non-maximum suppression (NMS). In this paper, we first analyze the
influence of NMS in modern real-time object detectors on inference speed, and
establish an end-to-end speed benchmark. To avoid the inference delay caused by
NMS, we propose a Real-Time DEtection TRansformer (RT-DETR), the first
real-time end-to-end object detector to our best knowledge. Specifically, we
design an efficient hybrid encoder to efficiently process multi-scale features
by decoupling the intra-scale interaction and cross-scale fusion, and propose
IoU-aware query selection to improve the initialization of object queries. In
addition, our proposed detector supports flexibly adjustment of the inference
speed by using different decoder layers without the need for retraining, which
facilitates the practical application of real-time object detectors. Our
RT-DETR-L achieves 53.0% AP on COCO val2017 and 114 FPS on T4 GPU, while
RT-DETR-X achieves 54.8% AP and 74 FPS, outperforming all YOLO detectors of the
same scale in both speed and accuracy. Furthermore, our RT-DETR-R50 achieves
53.1% AP and 108 FPS, outperforming DINO-Deformable-DETR-R50 by 2.2% AP in
accuracy and by about 21 times in FPS. Source code and pretrained models will
be available at PaddleDetection.

本文提出了第一款实时端到端物体检测器 ——Real-Time DEtection TRansformer（RT-DETR），支持灵活调整推理速度且在速度和准确度上均优于同一规模的 YOLO 检测器以及 DINO-Deformable-DETR-R50 模型，采用了高效的混合编码器处理多尺度特征并提出 IoU 感知的查询选择。

DETR 在实时目标检测中击败了 YOLO

DETRs Beat YOLOs on Real-time Object Detection

A naive application of state-of-the-art bidirectional encoders for streaming
sequence tagging would require encoding each token from scratch for each new
token in an incremental streaming input (like transcribed speech). The lack of
re-usability of previous computation leads to a higher number of Floating Point
Operations (or FLOPs) and higher number of unnecessary label flips. Increased
FLOPs consequently lead to higher wall-clock time and increased label flipping
leads to poorer streaming performance. In this work, we present a Hybrid
Encoder with Adaptive Restart (HEAR) that addresses these issues while
maintaining the performance of bidirectional encoders over the offline (or
complete) inputs while improving performance on streaming (or incomplete)
inputs. HEAR has a Hybrid unidirectional-bidirectional encoder architecture to
perform sequence tagging, along with an Adaptive Restart Module (ARM) to
selectively guide the restart of bidirectional portion of the encoder. Across
four sequence tagging tasks, HEAR offers FLOP savings in streaming settings
upto 71.1% and also outperforms bidirectional encoders for streaming
predictions by upto +10% streaming exact match.

该研究提出了一种名为 HEAR 的混合编码器，它具有自适应重启模块，可用于流式序列标记任务，能够在维持初始双向编码器性能的同时，提高流式输入的性能，并在节省高达 71.1% 的 FLOP 以及在准确度方面优于双向编码器 10% 的数据集上，得到良好的流式标记表现。