Generative AI and large language models hold great promise in enhancing
programming education by generating individualized feedback and hints for
learners. Recent works have primarily focused on improving the quality of
generated feedback to achieve human tutors' quality. While quality is an
important performance criterion, it is not the only criterion to optimize for
real-world educational deployments. In this paper, we benchmark language models
for programming feedback generation across several performance criteria,
including quality, cost, time, and data privacy. The key idea is to leverage
recent advances in the new paradigm of in-browser inference that allow running
these models directly in the browser, thereby providing direct benefits across
cost and data privacy. To boost the feedback quality of small models compatible
with in-browser inference engines, we develop a fine-tuning pipeline based on
GPT-4 generated synthetic data. We showcase the efficacy of fine-tuned
Llama3-8B and Phi3-3.8B 4-bit quantized models using WebLLM's in-browser
inference engine on three different Python programming datasets. We will
release the full implementation along with a web app and datasets to facilitate
further research on in-browser language models.

通过在浏览器中进行推理的新范式，基于生成技术的人工智能和大型语言模型能够在编程教育中生成个性化反馈和提示，本文通过基准测试语言模型在编程反馈生成中的质量、成本、时间和数据隐私等多个性能标准，展示了与浏览器推理引擎兼容的小型模型的反馈质量提升，并使用 WebLLM 的浏览器推理引擎在三个不同的 Python 编程数据集上展示了经过精调的 Llama3-8B 和 Phi3-3.8B 4 位量化模型的有效性，同时提供完整的实现、Web 应用和数据集以促进进一步的浏览器语言模型研究。

Hints-In-Browser：针对编程反馈生成的语言模型基准测试

Hints-In-Browser: Benchmarking Language Models for Programming Feedback  Generation

Deep Learning (DL) is increasingly being integrated into Web applications
through a method known as "in-browser inference", where the DL processes occur
directly within Web browsers. However, the actual performance of this method
and its effect on user experience quality (QoE) is not well-understood. This
gap in knowledge necessitates new forms of QoE measurement, going beyond
traditional metrics such as page load time. To address this, we conducted the
first extensive performance evaluation of in-browser inference. We introduced
new metrics for this purpose: responsiveness, smoothness, and inference
accuracy.
Our thorough study included 9 widely-used DL models and tested them across 50
popular PC Web browsers. The findings show a significant latency issue with
in-browser inference: it's on average 16.9 times slower on CPU and 4.9 times
slower on GPU than native inference methods. Several factors contribute to this
latency, including underused hardware instruction sets, inherent delays in the
runtime environment, resource competition within the browser, and
inefficiencies in software libraries and GPU abstractions.
Moreover, in-browser inference demands a lot of memory, sometimes up to 334.6
times more than the size of the DL models themselves. This excessive memory
usage is partly due to suboptimal memory management. Additionally, we noticed
that in-browser inference increases the time it takes for graphical user
interface (GUI) components to load in web browsers by a significant 67.2\%,
which severely impacts the overall QoE for users of web applications that
depend on this technology.

深度学习在 Web 应用中的 “浏览器内推理” 方法正在被广泛应用，然而该方法的实际性能以及对用户体验质量（QoE）的影响尚未得到很好的了解。本研究通过对广泛使用的 9 个深度学习模型在 50 个流行的 PC Web 浏览器上的测试，提出了新的性能指标，包括响应速度、流畅度和推理准确性，研究结果显示浏览器内推理存在显著的延迟问题，相比本地推理方法，CPU 平均延迟慢 16.9 倍，GPU 平均延迟慢 4.9 倍，并且浏览器内推理对内存的需求非常大，有时达到 DL 模型本身大小的 334.6 倍，这导致了内存管理的亚最优性。此外，浏览器内推理还显著增加了 Web 浏览器中图形用户界面（GUI）组件加载的时间，对依赖该技术的 Web 应用用户的整体 QoE 造成严重影响。