Recently, Large Langauge Models (LLMs) have gained a lot of attention in the
Software Engineering (SE) community. LLMs or their variants pre-trained on code
are used for many SE tasks. A main approach for adapting LLMs to the downstream
task is to fine-tune the models. However, with having billions-parameters-LLMs,
fine-tuning the models is not practical. An alternative approach is using
Parameter Efficient Fine Tuning (PEFT), in which the model parameters are
frozen and only a few added parameters are trained. Though the LLMs are used
for programming languages such as Python and Java widely, their capability for
low-resource languages is limited. In this work, we empirically study PEFT
methods, LoRA and Compacter, on CodeT5 and CodeLlama. We will assess their
performance compared to fully fine-tuned models, whether they can be used for
knowledge transfer from natural language models to code (using T5 and Llama
models), and their ability to adapt the learned knowledge to an unseen
language. For the unseen language, we aim to study R, as it has a wide
community. The adaptability with less computational costs makes LLMs accessible
in scenarios where heavy computational resources are not available. Moreover,
studying R opens new opportunities for using LLMs for other languages. We
anticipate our findings to showcase the capabilities of PEFT for code LLMs for
R and reveal the improvement areas.

通过实证研究 PEFT 方法（LoRA 和 Compacter）在 CodeT5 和 CodeLlama 上的表现，评估它们相较于完全微调模型的性能、是否可用于从自然语言模型到代码的知识转移，以及它们适应学习知识到一种未见语言的能力。我们的研究目标是研究 R 语言，其具有广泛的社区，而 LLMs 的适应性和较低的计算成本使其能够在缺乏大量计算资源的情况下使用，此外，研究 R 语言为使用 LLMs 于其他语言提供了新机会，我们期望得出 PEFT 方法在 R 语言的代码 LLMs 上的能力以及改进领域。

代码大语言模型参数高效方法的实证研究及其向 R 的知识转移

Empirical Studies of Parameter Efficient Methods for Large Language  Models of Code and Knowledge Transfer to R

This paper reports an evaluation of ChatGPT's capability of generating R
programming language code from natural language input. A dataset specially
designed for generating R program code was constructed with metadata to support
scenario-based testing and evaluation of code generation capabilities in
various usage scenarios of different levels of difficulty and different types
of programs. The evaluation takes a multiple attempt process in which the
tester tries to complete the code generation task through a number of attempts
until a satisfactory solution is obtained or gives up after a fixed number of
maximal attempts. In each attempt the tester formulates a natural language
input to ChatGPT based on the previous results and the task to be completed. In
addition to the metrics of average numbers of attempts and average amount of
time taken to complete the tasks, the final generated solutions are then
assessed on a number of quality attributes, including accuracy, completeness,
conciseness, readability, well structuredness, logic clarity, depth of
ex-planation, and coverage of parameters. Our experiments demonstrated that
ChatGPT is in general highly capable of generating high quality R program code
as well as textual explanations although it may fail on hard programming tasks.
The experiment data also shows that human developers can hardly learn from
experiences naturally to improve the skill of using ChatGPT to generate code.

对 ChatGPT 的 R 编程语言代码生成能力进行评估，使用特别设计的数据集进行测试和评估，结果显示 ChatGPT 能够生成高质量的 R 程序代码和文本解释，但在复杂任务中可能会失败。同时，实验结果表明人类开发者难以自然地从经验中学习，以提高使用 ChatGPT 生成代码的技能。