While large code datasets have become available in recent years, acquiring
representative training data with full coverage of general code distribution
remains challenging due to the compositional nature of code and the complexity
of software. This leads to the out-of-distribution (OOD) issues with unexpected
model inference behaviors that have not been systematically studied yet. We
contribute the first systematic approach that simulates various OOD scenarios
along different dimensions of data properties and investigates the model
behaviors in such scenarios. Our extensive studies on six state-of-the-art
models for three code generation tasks expose several failure modes caused by
the out-of-distribution issues. It thereby provides insights and sheds light
for future research in terms of generalization, robustness, and inductive
biases of source code models.

由于代码的组合性和软件的复杂性，获取具有完整代码分布覆盖的代表性训练数据仍然具有挑战性。因此，我们提出了第一种系统性方法，模拟不同维度数据属性的各种 OOD 场景，并研究在这些场景下模型的行为。我们对三项代码生成任务中的六种最先进的模型进行了广泛的研究，发现了一些由 OOD 问题引起的失败模式。此研究具有启示作用，为未来关于源代码模型的泛化、鲁棒性和归纳偏差等的研究提供了新的视角。