In behavioural testing, system functionalities underrepresented in the
standard evaluation setting (with a held-out test set) are validated through
controlled input-output pairs. Optimising performance on the behavioural tests
during training (behavioural learning) would improve coverage of phenomena not
sufficiently represented in the i.i.d. data and could lead to seemingly more
robust models. However, there is the risk that the model narrowly captures
spurious correlations from the behavioural test suite, leading to
overestimation and misrepresentation of model performance -- one of the
original pitfalls of traditional evaluation. In this work, we introduce BeLUGA,
an analysis method for evaluating behavioural learning considering
generalisation across dimensions of different granularity levels. We optimise
behaviour-specific loss functions and evaluate models on several partitions of
the behavioural test suite controlled to leave out specific phenomena. An
aggregate score measures generalisation to unseen functionalities (or
overfitting). We use BeLUGA to examine three representative NLP tasks
(sentiment analysis, paraphrase identification and reading comprehension) and
compare the impact of a diverse set of regularisation and domain generalisation
methods on generalisation performance.

介绍了一种行为学习分析方法 BeLUGA，评估针对不同粒度级别的维度的泛化以及优化行为特定的损失函数并比较不同的正则化和域泛化方法对于 NLP 任务泛化性能的影响。