Transformers have excelled in many tasks including vision. However, efficient
deployment of transformer models in low-latency or high-throughput applications
is hindered by the computation in the attention mechanism which involves
expensive operations such as matrix multiplication and Softmax. To address
this, we introduce ReduceFormer, a family of models optimized for efficiency
with the spirit of attention. ReduceFormer leverages only simple operations
such as reduction and element-wise multiplication, leading to greatly
simplified architecture and improved inference performance, with up to 37%
reduction in latency and 44% improvement in throughput, while maintaining
competitive accuracy comparable to other recent methods. The proposed model
family is suitable for edge devices where compute resource and memory bandwidth
are limited, as well as for cloud computing where high throughput is sought
after.

Transformers 的计算机视觉任务中，通过使用简化架构和改进的推理性能的 ReduceFormer 模型系列，有效地解决了传统 Transformer 模型中的计算问题，降低了延迟和提高了吞吐量，适用于计算资源和内存带宽有限的边缘设备以及追求高吞吐量的云计算。