In this technical report, we present three novel datasets of Kotlin code:
KStack, KStack-clean, and KExercises. We also describe the results of
fine-tuning CodeLlama and DeepSeek models on this data. Additionally, we
present a version of the HumanEval benchmark rewritten by human experts into
Kotlin - both the solutions and the tests. Our results demonstrate that small,
high-quality datasets (KStack-clean and KExercises) can significantly improve
model performance on code generation tasks, achieving up to a 16-point increase
in pass rate on the HumanEval benchmark. Lastly, we discuss potential future
work in the field of improving language modeling for Kotlin, including the use
of static analysis tools in the learning process and the introduction of more
intricate and realistic benchmarks.

我们介绍了三个新的 Kotlin 代码数据集，KStack，KStack-clean 和 KExercises。我们描述了在这些数据上对 CodeLlama 和 DeepSeek 模型的微调结果。我们还展示了人工专家将 HumanEval 基准测试改写为 Kotlin 的版本，包括解决方案和测试。我们的结果表明，小而高质量的数据集（KStack-clean 和 KExercises）可以显着提高模型在代码生成任务上的性能，在 HumanEval 基准测试中通过率提高了最多 16 个百分点。最后，我们讨论了改进 Kotlin 语言建模的潜在未来工作，包括在学习过程中使用静态分析工具和引入更复杂、更实际的基准测试。

Kotlin ML Pack: 技术报告

Kotlin ML Pack: Technical Report

Large Language Models (LLMs) have been widely employed in programming
language analysis to enhance human productivity. Yet, their reliability can be
compromised by various code distribution shifts, leading to inconsistent
outputs. While probabilistic methods are known to mitigate such impact through
uncertainty calibration and estimation, their efficacy in the language domain
remains underexplored compared to their application in image-based tasks. In
this work, we first introduce a large-scale benchmark dataset, incorporating
three realistic patterns of code distribution shifts at varying intensities.
Then we thoroughly investigate state-of-the-art probabilistic methods applied
to CodeLlama using these shifted code snippets. We observe that these methods
generally improve the uncertainty awareness of CodeLlama, with increased
calibration quality and higher uncertainty estimation~(UE) precision. However,
our study further reveals varied performance dynamics across different criteria
(e.g., calibration error vs misclassification detection) and trade-off between
efficacy and efficiency, highlighting necessary methodological selection
tailored to specific contexts.

大规模语言模型在编程语言分析中被广泛应用以提高人类生产力。本研究通过引入大规模基准数据集，调查了应用概率方法来处理与代码分布变化相关的问题，结果表明这些方法能够提高 CodeLlama 的不确定性感知能力，并增加校准质量和不确定性估计精度。然而，在不同标准（如校准误差与错误分类检测）和效能与效率之间存在不同的性能动态，强调了需根据特定环境进行方法选择的必要性。