Personalized chatbot-based teaching assistants can be crucial in addressing
increasing classroom sizes, especially where direct teacher presence is
limited. Large language models (LLMs) offer a promising avenue, with increasing
research exploring their educational utility. However, the challenge lies not
only in establishing the efficacy of LLMs but also in discerning the nuances of
interaction between learners and these models, which impact learners'
engagement and results. We conducted a formative study in an undergraduate
computer science classroom (N=145) and a controlled experiment on Prolific
(N=356) to explore the impact of four pedagogically informed guidance
strategies and the interaction between student approaches and LLM responses.
Direct LLM answers marginally improved performance, while refining student
solutions fostered trust. Our findings suggest a nuanced relationship between
the guidance provided and LLM's role in either answering or refining student
input. Based on our findings, we provide design recommendations for optimizing
learner-LLM interactions.

个性化聊天机器人教学助手在解决教室规模扩大的问题上至关重要，特别是在教师直接存在有限的情况下。大型语言模型（LLM）提供了一个有前途的途径，不断有研究探索它们在教育上的实用性。然而，挑战不仅在于建立 LLM 的效能，而且还在于理解学习者与这些模型之间的互动细微差别，这会影响学习者的参与和结果。我们在一个本科计算机科学课堂（N=145）进行了一个形成性研究，并在 Prolific（N=356）上进行了一项受控实验，探索四种教学指导策略的影响以及学生方法与 LLM 响应之间的互动关系。直接的 LLM 答案稍微提高了表现，而改进学生解决方案培养了信任。我们的发现表明，所提供的指导和 LLM 在回答或改进学生输入方面的角色之间存在微妙的关系。根据我们的发现，我们提供了优化学习者 - LLM 互动的设计建议。