As a widely used non-linear activation, Rectified Linear Unit (ReLU)
separates noise and signal in a feature map by learning a threshold or bias.
However, we argue that the classification of noise and signal not only depends
on the magnitude of responses, but also the context of how the feature
responses would be used to detect more abstract patterns in higher layers. In
order to output multiple response maps with magnitude in different ranges for a
particular visual pattern, existing networks employing ReLU and its variants
have to learn a large number of redundant filters. In this paper, we propose a
multi-bias non-linear activation (MBA) layer to explore the information hidden
in the magnitudes of responses. It is placed after the convolution layer to
decouple the responses to a convolution kernel into multiple maps by
multi-thresholding magnitudes, thus generating more patterns in the feature
space at a low computational cost. It provides great flexibility of selecting
responses to different visual patterns in different magnitude ranges to form
rich representations in higher layers. Such a simple and yet effective scheme
achieves the state-of-the-art performance on several benchmarks.

本文提出了一种名为 MBA 的多偏置非线性激活层，它采用多重阈值对卷积核的响应进行解耦，从而在特征空间中生成更多的模式，并在低计算成本下选择不同幅度范围内的响应来形成更丰富的表示。此简单而有效的方案在多个基准测试中实现了最先进的性能。