The question of whether deep neural networks are good at generalising beyond
their immediate training experience is of critical importance for
learning-based approaches to AI. Here, we consider tests of out-of-sample
generalisation that require an agent to respond to never-seen-before
instructions by manipulating and positioning objects in a 3D Unity simulated
room. We first describe a comparatively generic agent architecture that
exhibits strong performance on these tests. We then identify three aspects of
the training regime and environment that make a significant difference to its
performance: (a) the number of object/word experiences in the training set; (b)
the visual invariances afforded by the agent's perspective, or frame of
reference; and (c) the variety of visual input inherent in the perceptual
aspect of the agent's perception. Our findings indicate that the degree of
generalisation that networks exhibit can depend critically on particulars of
the environment in which a given task is instantiated. They further suggest
that the propensity for neural networks to generalise in systematic ways may
increase if, like human children, those networks have access to many frames of
richly varying, multi-modal observations as they learn.

研究神经网络的泛化能力，通过 3D Unity 模拟环境测试网络在未见过的情况下的表现并发现了训练、环境的细节影响泛化能力，建议在网络训练时按照孩子多感官、多模态的观察方式输入信息。