Voice plays an important role in our lives by facilitating communication,
conveying emotions, and indicating health. Therefore, tracking vocal
interactions can provide valuable insight into many aspects of our lives. This
paper presents our ongoing efforts to design a new vocal tracking system we
call VoCopilot. VoCopilot is an end-to-end system centered around an
energy-efficient acoustic hardware and firmware combined with advanced machine
learning models. As a result, VoCopilot is able to continuously track
conversations, record them, transcribe them, and then extract useful insights
from them. By utilizing large language models, VoCopilot ensures the user can
extract useful insights from recorded interactions without having to learn
complex machine learning techniques. In order to protect the privacy of end
users, VoCopilot uses a novel wake-up mechanism that only records conversations
of end users. Additionally, all the rest of pipeline can be run on a commodity
computer (Mac Mini M2). In this work, we show the effectiveness of VoCopilot in
real-world environment for two use cases.

本文介绍了我们正在进行的设计一种名为 VoCopilot 的新型声音跟踪系统的努力，它是一个以节能的声学硬件和固件为中心，结合先进的机器学习模型的端到端系统。通过利用大型语言模型，VoCopilot 能够在不学习复杂的机器学习技术的情况下，持续跟踪对话，记录并转录它们，并从中提取有用的见解。为了保护终端用户的隐私，VoCopilot 使用一种新颖的唤醒机制，只记录终端用户的对话。此外，整个处理过程可以在一台普通的计算机上运行（Mac Mini M2）。在本工作中，我们展示了 VoCopilot 在两个使用场景的真实环境中的有效性。