The expressiveness of neural networks highly depends on the nature of the
activation function, although these are usually assumed predefined and fixed
during the training stage. In this paper we present Expressive Neural Network
(ENN), a novel architecture in which the non-linear activation functions are
modeled using the Discrete Cosine Transform (DCT) and adapted using
backpropagation during training. This parametrization keeps the number of
trainable parameters low, is appropriate for gradient-based schemes, and adapts
to different learning tasks. This is the first non-linear model for activation
functions that relies on a signal processing perspective, providing high
flexibility and expressiveness to the network. We contribute with insights in
the explainability of the network at convergence by recovering the concept of
bump, this is, the response of each activation function in the output space to
provide insights. Finally, through exhaustive experiments we show that the
model can adapt to classification and regression tasks. The performance of ENN
outperforms state of the art benchmarks, providing up to a 40\% gap in accuracy
in some scenarios.

该研究提出了一种新的神经网络架构 - Expressive Neural Network (ENN)，其中非线性激活函数使用离散余弦变换 (DCT) 建模，并在训练过程中使用反向传播进行自适应，实现了较高的灵活性和表现力，同时可以适应不同的学习任务。经过大量实验证明，ENN 的性能超过当前的最先进技术，并在某些场景下提供高达 40％的准确性差距。