Language models are promising solutions for tackling increasing complex problems. In software engineering, they recently attracted attention in code assistants, with programs automatically written in a given programming language from a programming task description in natural language. They have the potential to save time and effort when writing code. However, these systems are currently poorly understood, preventing them from being used optimally. In this paper, we investigate the various input parameters of two language models, and conduct a study to understand if variations of these input parameters (e.g. programming task description and the surrounding context, creativity of the language model, number of generated solutions) can have a significant impact on the quality of the generated programs. We design specific operators for varying input parameters and apply them over two code assistants (Copilot and Codex) and two benchmarks representing algorithmic problems (HumanEval and LeetCode). Our results showed that varying the input parameters can significantly improve the performance of language models. However, there is a tight dependency when varying the temperature, the prompt and the number of generated solutions, making potentially hard for developers to properly control the parameters to obtain an optimal result. This work opens opportunities to propose (automated) strategies for improving performance.

本文研究了两种语言模型的各种输入参数，并进行了研究以了解这些输入参数的变化（例如编程任务描述和周围环境，语言模型的创造力，生成的解决方案数量）是否会对生成的程序的质量产生显着影响。研究结果表明，改变输入参数可以显着提高语言模型的性能。但是，当改变温度、提示和生成的解决方案数量时，存在严格的依赖关系，可能很难让开发人员正确控制参数以获得最佳结果。

试点Copilot和Codex：高温，低提示，还是黑魔法？