Language User Interfaces (LUIs) could improve human-machine interaction for a
wide variety of tasks, such as playing music, getting insights from databases,
or instructing domestic robots. In contrast to traditional hand-crafted
approaches, recent work attempts to build LUIs in a data-driven way using
modern deep learning methods. To satisfy the data needs of such learning
algorithms, researchers have constructed benchmarks that emphasize the quantity
of collected data at the cost of its naturalness and relevance to real-world
LUI use cases. As a consequence, research findings on such benchmarks might not
be relevant for developing practical LUIs. The goal of this paper is to
bootstrap the discussion around this issue, which we refer to as the
benchmarks' low ecological validity. To this end, we describe what we deem an
ideal methodology for machine learning research on LUIs and categorize five
common ways in which recent benchmarks deviate from it. We give concrete
examples of the five kinds of deviations and their consequences. Lastly, we
offer a number of recommendations as to how to increase the ecological validity
of machine learning research on LUIs.

使用近代深度学习方法构建 Data-driven 的语言用户界面（LUIs），作者讨论了生成的数据来源对于实际应用的有效性的影响，指出当前一些基准测试对真实应用存在差距，提出了提高 LUIs 机器学习研究的生态效度的建议。