SoftMax is a ubiquitous ingredient of modern machine learning algorithms. It
maps an input vector onto a probability simplex and reweights the input by
concentrating the probability mass at large entries. Yet, as a smooth
approximation to the Argmax function, a significant amount of probability mass
is distributed to other, residual entries, leading to poor interpretability and
noise. Although sparsity can be achieved by a family of SoftMax variants, they
often require an alternative loss function and do not preserve multi-modality.
We show that this trade-off between multi-modality and sparsity limits the
expressivity of SoftMax as well as its variants. We provide a solution to this
tension between objectives by proposing a piece-wise differentiable function,
termed MultiMax, which adaptively modulates the output distribution according
to input entry range. Through comprehensive analysis and evaluation, we show
that MultiMax successfully produces a distribution that supresses irrelevant
entries while preserving multimodality, with benefits in image classification,
language modeling and machine translation. The code is available at
this https URL ZhouYuxuanYX/MultiMax.

通过提出 MultiMax 这种分段可微凸函数，根据输入条目范围自适应调节输出分布，我们解决了 SoftMax 及其变体在多模态和稀疏性之间的权衡问题，成功产生了抑制无关条目而保留多模态的分布，对图像分类、语言建模和机器翻译产生了有益的影响。

MultiMax：稀疏和多模态的注意力学习

MultiMax: Sparse and Multi-Modal Attention Learning

We introduce the use of rectified linear units (ReLU) as the classification
function in a deep neural network (DNN). Conventionally, ReLU is used as an
activation function in DNNs, with Softmax function as their classification
function. However, there have been several studies on using a classification
function other than Softmax, and this study is an addition to those. We
accomplish this by taking the activation of the penultimate layer $h_{n - 1}$
in a neural network, then multiply it by weight parameters $\theta$ to get the
raw scores $o_{i}$. Afterwards, we threshold the raw scores $o_{i}$ by $0$,
i.e. $f(o) = \max(0, o_{i})$, where $f(o)$ is the ReLU function. We provide
class predictions $\hat{y}$ through argmax function, i.e. argmax $f(x)$.

本研究介绍了在深度神经网络中使用修正线性单元作为分类函数来预测类别，通过将神经网络中的倒数第二层的输出与权重参数相乘得到原始分数，再使用修正线性单元函数进行阈值处理，最后通过 argmax 函数得到分类预测。