Language model-based code completion models have quickly grown in use,
helping thousands of developers write code in many different programming
languages. However, research on code completion models typically focuses on
imperative languages such as Python and JavaScript, which results in a lack of
representation for functional programming languages. Consequently, these models
often perform poorly on functional languages such as Haskell. To investigate
whether this can be alleviated, we evaluate the performance of two language
models for code, CodeGPT and UniXcoder, on the functional programming language
Haskell. We fine-tune and evaluate the models on Haskell functions sourced from
a publicly accessible Haskell dataset on HuggingFace. Additionally, we manually
evaluate the models using our novel translated HumanEval dataset. Our automatic
evaluation shows that knowledge of imperative programming languages in the
pre-training of LLMs may not transfer well to functional languages, but that
code completion on functional languages is feasible. Consequently, this shows
the need for more high-quality Haskell datasets. A manual evaluation on
HumanEval-Haskell indicates CodeGPT frequently generates empty predictions and
extra comments, while UniXcoder more often produces incomplete or incorrect
predictions. Finally, we release HumanEval-Haskell, along with the fine-tuned
models and all code required to reproduce our experiments on GitHub
(this https URL).

评估了两个语言模型（CodeGPT 和 UniXcoder）在函数式编程语言 Haskell 上的性能表现，发现在函数式编程语言上代码补全是可行的，但存在空预测、额外注释和不完整 / 错误预测的问题，因此需要更多高质量的 Haskell 数据集。