This paper addresses spoken language understanding (SLU) on
microcontroller-like embedded devices, integrating on-device execution with
cloud offloading in a novel fashion. We exploit temporal locality in a device's
speech inputs and accordingly reuse recent SLU inferences. Our idea is simple:
let the device match new inputs against cached results, and only offload
unmatched inputs to the cloud for full inference. Realization of this idea,
however, is non-trivial: the device needs to compare acoustic features in a
robust, low-cost way. To this end, we present XYZ, a speech cache for tiny
devices. It matches speech inputs at two levels of representations: first by
clustered sequences of raw sound units, then as sequences of phonemes. Working
in tandem, the two representations offer complementary cost/accuracy tradeoffs.
To further boost accuracy, our cache is learning: with the mismatched and then
offloaded inputs, it continuously finetunes the device's feature extractors
(with the assistance of the cloud). We implement XYZ on an off-the-shelf STM32
microcontroller. The resultant implementation has a small memory footprint of
2MB. Evaluated on challenging speech benchmarks, our system resolves 45%--90%
of inputs on device, reducing the average latency by up to 80% compared to
offloading to popular cloud speech services. Our benefit is pronounced even in
adversarial settings -- noisy environments, cold cache, or one device shared by
a number of users.

探讨在类微控制器嵌入式设备上的口语理解（SLU），将设备上的执行与云端卸载结合在一起。通过在设备的语音输入中利用时间局部性，重复使用最近的 SLU 推断。我们的想法很简单：让设备将新输入与缓存结果进行匹配，并只将匹配不成功的输入卸载到云端进行完整的推断。实现这个想法，然而，是非常复杂的：设备需要以稳健、低成本的方式比较声学特征。为了达到这个目的，我们提出了 XYZ，一种适用于微型设备的语音缓存。它通过聚类的原始声音单元序列和音素序列两个级别来匹配语音输入，两个表示方法相辅相成，提供了成本 / 准确性的互补权衡。为了进一步提高准确性，我们的缓存是可以学习的：通过不匹配的以及卸载到云端的输入，它不断地微调设备的特征提取器（在云端的辅助下）。我们在一款现成的 STM32 微控制器上实现了 XYZ。实现的结果具有 2MB 的小内存占用量。在具有挑战性的语音基准测试中进行评估时，我们的系统能够在设备上解决 45% 至 90% 的输入，与卸载到流行的云语音服务相比，平均延迟降低了高达 80%。即使在对抗性环境（噪声环境、冷缓存或多个用户共享设备）中，我们的优势仍然显著。