We provide an overview of several non-linear activation functions in a neural
network architecture that have proven successful in many machine learning
applications. We conduct an empirical analysis on the effectiveness of using
these function on the MNIST classification task, with the aim of clarifying
which functions produce the best results overall. Based on this first set of
results, we examine the effects of building deeper architectures with an
increasing number of hidden layers. We also survey the impact of using, on the
same task, different initialisation schemes for the weights of our neural
network. Using these sets of experiments as a base, we conclude by providing a
optimal neural network architecture that yields impressive results in accuracy
on the MNIST classification task.

本研究总结了神经网络架构中几种非线性激活函数的使用情况，并对这些函数在 MNIST 分类任务上的效果进行了实证分析，以确定哪些函数能够产生最佳结果。基于这些结果，研究了构建具有不同数量隐藏层的深度架构以及使用不同的初始化方案对神经网络的影响。最终提供了一个最优的神经网络架构，可在 MNIST 分类任务中获得令人印象深刻的准确度。