This research paper introduces an innovative AI coaching approach by
integrating vision-encoder-decoder models. The feasibility of this method is
demonstrated using a Vision Transformer as the encoder and GPT-2 as the
decoder, achieving a seamless integration of visual input and textual
interaction. Departing from conventional practices of employing distinct models
for image recognition and text-based coaching, our integrated architecture
directly processes input images, enabling natural question-and-answer dialogues
with the AI coach. This unique strategy simplifies model architecture while
enhancing the overall user experience in human-AI interactions. We showcase
sample results to demonstrate the capability of the model. The results
underscore the methodology's potential as a promising paradigm for creating
efficient AI coach models in various domains involving visual inputs.
Importantly, this potential holds true regardless of the particular visual
encoder or text decoder chosen. Additionally, we conducted experiments with
different sizes of GPT-2 to assess the impact on AI coach performance,
providing valuable insights into the scalability and versatility of our
proposed methodology.

本研究论文介绍了一种创新的 AI 辅导方法，通过整合视觉编码器 - 解码器模型，以 Vision Transformer 作为编码器，GPT-2 作为解码器，实现了对视觉输入和文本交互的无缝集成。与传统实践中使用不同模型进行图像识别和文本辅导的做法不同，我们的集成架构直接处理输入图像，实现与 AI 辅导员的自然问答对话。这种独特的策略简化了模型架构，同时提升了人工智能与人类之间的整体用户体验。我们展示了样本结果以证明该模型的能力。结果强调了该方法作为创建涉及视觉输入的多领域高效 AI 辅导模型的潜力，尤其重要的是，无论选择了哪种视觉编码器或文本解码器，该潜力都是成立的。此外，我们对不同大小的 GPT-2 进行了实验，评估其对 AI 辅导员性能的影响，为我们所提出的方法的可扩展性和多功能性提供了宝贵的见解。