We present a deep layered architecture that generalizes convolutional neural
networks (ConvNets). The architecture, called SimNets, is driven by two
operators: (i) a similarity function that generalizes inner-product, and (ii) a
log-mean-exp function called MEX that generalizes maximum and average. The two
operators applied in succession give rise to a standard neuron but in "feature
space". The feature spaces realized by SimNets depend on the choice of the
similarity operator. The simplest setting, which corresponds to a convolution,
realizes the feature space of the Exponential kernel, while other settings
realize feature spaces of more powerful kernels (Generalized Gaussian, which
includes as special cases RBF and Laplacian), or even dynamically learned
feature spaces (Generalized Multiple Kernel Learning). As a result, the SimNet
contains a higher abstraction level compared to a traditional ConvNet. We argue
that enhanced expressiveness is important when the networks are small due to
run-time constraints (such as those imposed by mobile applications). Empirical
evaluation validates the superior expressiveness of SimNets, showing a
significant gain in accuracy over ConvNets when computational resources at
run-time are limited. We also show that in large-scale settings, where
computational complexity is less of a concern, the additional capacity of
SimNets can be controlled with proper regularization, yielding accuracies
comparable to state of the art ConvNets.

SimNets 是一种深度分层架构，在类卷积神经网络的基础上引入了相似函数与对数均值指数函数，从而实现了更强的表达能力和更广泛的特征空间，并且在计算资源受限的场景下具有显著的优势。

深度 SimNets

Deep SimNets

We present a deep layered architecture that generalizes classical
convolutional neural networks (ConvNets). The architecture, called SimNets, is
driven by two operators, one being a similarity function whose family contains
the convolution operator used in ConvNets, and the other is a new soft
max-min-mean operator called MEX that realizes classical operators like ReLU
and max pooling, but has additional capabilities that make SimNets a powerful
generalization of ConvNets. Three interesting properties emerge from the
architecture: (i) the basic input to hidden layer to output machinery contains
as special cases kernel machines with the Exponential and Generalized Gaussian
kernels, the output units being "neurons in feature space" (ii) in its general
form, the basic machinery has a higher abstraction level than kernel machines,
and (iii) initializing networks using unsupervised learning is natural.
Experiments demonstrate the capability of achieving state of the art accuracy
with networks that are an order of magnitude smaller than comparable ConvNets.

本文提出了一种称为 SimNets 的深度分层体系结构，可泛化经典的卷积神经网络（ConvNets），使用相似度函数和新的 soft max-min-mean 操作器 MEX 进行驱动，并且可以很自然地使用无监督学习进行网络初始化，并且实验表明，使用比可比的 ConvNets 小一个数量级的网络可以获得最先进的准确性。