Language model-based code completion models have quickly grown in use,
helping thousands of developers write code in many different programming
languages. However, research on code completion models typically focuses on
imperative languages such as Python and JavaScript, which results in a lack of
representation for functional programming languages. Consequently, these models
often perform poorly on functional languages such as Haskell. To investigate
whether this can be alleviated, we evaluate the performance of two language
models for code, CodeGPT and UniXcoder, on the functional programming language
Haskell. We fine-tune and evaluate the models on Haskell functions sourced from
a publicly accessible Haskell dataset on HuggingFace. Additionally, we manually
evaluate the models using our novel translated HumanEval dataset. Our automatic
evaluation shows that knowledge of imperative programming languages in the
pre-training of LLMs may not transfer well to functional languages, but that
code completion on functional languages is feasible. Consequently, this shows
the need for more high-quality Haskell datasets. A manual evaluation on
HumanEval-Haskell indicates CodeGPT frequently generates empty predictions and
extra comments, while UniXcoder more often produces incomplete or incorrect
predictions. Finally, we release HumanEval-Haskell, along with the fine-tuned
models and all code required to reproduce our experiments on GitHub
(this https URL).

评估了两个语言模型（CodeGPT 和 UniXcoder）在函数式编程语言 Haskell 上的性能表现，发现在函数式编程语言上代码补全是可行的，但存在空预测、额外注释和不完整 / 错误预测的问题，因此需要更多高质量的 Haskell 数据集。

探究语言模型在函数式编程语言中完成代码的性能：一个 Haskell 案例研究

Investigating the Performance of Language Models for Completing Code in  Functional Programming Languages: a Haskell Case Study

Program synthesis is the process of generating a computer program following a
set of specifications, which can be a high-level description of the problem
and/or a set of input-output examples. The synthesis can be modeled as a search
problem in which the search space is the set of all the programs valid under a
grammar. As the search space is vast, brute force is usually not viable and
search heuristics, such as genetic programming, also have difficulty navigating
it without any guidance. In this paper we present HOTGP, a new genetic
programming algorithm that synthesizes pure, typed, and functional programs.
HOTGP leverages the knowledge provided by the rich data-types associated with
the specification and the built-in grammar to constrain the search space and
improve the performance of the synthesis. The grammar is based on Haskell's
standard base library (the synthesized code can be directly compiled using any
standard Haskell compiler) and includes support for higher-order functions,
$\lambda$-functions, and parametric polymorphism. Experimental results show
that, when compared to $6$ state-of-the-art algorithms using a standard set of
benchmarks, HOTGP is competitive and capable of synthesizing the correct
programs more frequently than any other of the evaluated algorithms.

本文介绍了一种新的基于 Haskell 的基因编程算法 HOTGP，它利用规约的数据类型来约束搜索空间，更好地合成类型良好的纯函数编程。经验结果表明，与其他六种算法相比，HOTGP 在综合正确程序的能力上更为优异。

HOTGP -- 高阶类型遗传规划

HOTGP -- Higher-Order Typed Genetic Programming

We explore an approach to type-directed program synthesis rooted in
constraint-based type inference techniques. By doing this, we aim to more
efficiently synthesize polymorphic code while also tackling advanced typing
features such as GADTs that build upon polymorphism. Along the way, we also
present an implementation of these techniques in Scythe, a prototype live,
type-directed programming tool for the Haskell programming language and reflect
on our initial experience with the tool.

使用基于约束的类型推断技术探索了一种类型导向的程序综合方法，以更有效地综合多态代码，并解决建立在多态上的高级类型功能，如 GADTs。辅以我们在 Haskell 编程语言的原型工具 Scythe 中实现的这些技术，并反思我们的初步使用经验。