Achieving efficient and robust multi-channel data learning is a challenging
task in data science. By exploiting low-rankness in the transformed domain,
i.e., transformed low-rankness, tensor Singular Value Decomposition (t-SVD) has
achieved extensive success in multi-channel data representation and has
recently been extended to function representation such as Neural Networks with
t-product layers (t-NNs). However, it still remains unclear how t-SVD
theoretically affects the learning behavior of t-NNs. This paper is the first
to answer this question by deriving the upper bounds of the generalization
error of both standard and adversarially trained t-NNs. It reveals that the
t-NNs compressed by exact transformed low-rank parameterization can achieve a
sharper adversarial generalization bound. In practice, although t-NNs rarely
have exactly transformed low-rank weights, our analysis further shows that by
adversarial training with gradient flow (GF), the over-parameterized t-NNs with
ReLU activations are trained with implicit regularization towards transformed
low-rank parameterization under certain conditions. We also establish
adversarial generalization bounds for t-NNs with approximately transformed
low-rank weights. Our analysis indicates that the transformed low-rank
parameterization can promisingly enhance robust generalization for t-NNs.

探究 Tensor Singular Value Decomposition (t-SVD) 理论上对于 Neural Networks with t-product layers (t-NNs) 的影响，研究发现使用 exact transformed low-rank parameterization 的 t-NNs 能够提高其对于 adversarial generalization 的性能，并且即使 t-NNs 很少有完全 transformed low-rank weights，也可以通过 gradient flow (GF) 的 adversarial training 得到类似的效果，表明 transformed low-rank parameterization 在一定条件下可以提高 t-NNs 的鲁棒泛化性能。