Understanding theoretical properties of deep and locally connected nonlinear network, such as deep convolutional neural network (DCNN), is still a hard problem despite its empirical success. In this paper, we propose a novel theoretical framework for such networks with ReLU nonlinearity. The framework explicitly formulates data distribution, favors disentangled representations and is compatible with common regularization techniques such as Batch Norm. The framework is built upon teacher-student setting, by expanding the student forward/backward propagation onto the teacher's computational graph. The resulting model does not impose unrealistic assumptions (e.g., Gaussian inputs, independence of activation, etc). Our framework could help facilitate theoretical analysis of many practical issues, e.g. overfitting, generalization, disentangled representations in deep networks.

在深度卷积神经网络中，通过提出一种以带有ReLU非线性激活的网络为基础的新型理论框架，该框架通过在教师-学生设置中扩展学生的向前/向后传播，明确了数据分布，强调了分解表示，并且兼容常见的规则化技术，不会强加不现实的假设，这种框架有助于促进许多实用问题（如过拟合，概括，深度网络中的分解表示）的理论分析。

深度本地连接ReLU网络的理论框架