The paper introduces Hands-Free VR, a voice-based natural-language interface for VR. The user gives a command using their voice, the speech audio data is converted to text using a speech-to-text deep learning model that is fine-tuned for robustness to word phonetic similarity and to spoken English accents, and the text is mapped to an executable VR command using a large language model that is robust to natural language diversity. Hands-Free VR was evaluated in a controlled within-subjects study (N = 22) that asked participants to find specific objects and to place them in various configurations. In the control condition participants used a conventional VR user interface to grab, carry, and position the objects using the handheld controllers. In the experimental condition participants used Hands-Free VR. The results confirm that: (1) Hands-Free VR is robust to spoken English accents, as for 20 of our participants English was not their first language, and to word phonetic similarity, correctly transcribing the voice command 96.71% of the time; (2) Hands-Free VR is robust to natural language diversity, correctly mapping the transcribed command to an executable command in 97.83% of the time; (3) Hands-Free VR had a significant efficiency advantage over the conventional VR interface in terms of task completion time, total viewpoint translation, total view direction rotation, and total left and right hand translations; (4) Hands-Free VR received high user preference ratings in terms of ease of use, intuitiveness, ergonomics, reliability, and desirability.

该研究介绍了一种基于语音的自然语言界面，适用于虚拟现实。通过使用经过微调以适应语音相似性和英语口音的深度学习语音转文本模型，将语音音频数据转换为文本，并使用能够适应自然语言多样性的大型语言模型将文本映射为可执行的虚拟现实命令。在一项对22名参与者进行的被试内研究中，对Hands-Free VR进行了评估，要求参与者找到特定对象并按照不同配置进行放置。结果确认：（1）Hands-Free VR对于非以英语为母语的参与者和语音相似性具有鲁棒性，语音命令识别正确率达到96.71%；（2）Hands-Free VR对于自然语言多样性具有鲁棒性，97.83%的命令正确映射为可执行命令；（3）Hands-Free VR在任务完成时间、视点转换次数、视角旋转次数以及左右手移动次数方面与传统虚拟现实界面相比具有显著的效率优势；（4）Hands-Free VR在易用性、直观性、人体工程学、可靠性和期望度方面获得了较高的用户偏好评级。

免持操作虚拟现实