The distributed inference paradigm enables the computation workload to be distributed across multiple devices, facilitating the implementations of deep learning based intelligent services on extremely resource-constrained Internet of Things (IoT) scenarios. Yet it raises great challenges to perform complicated inference tasks relying on a cluster of IoT devices that are heterogeneous in their computing/communication capacity and prone to crash or timeout failures. In this paper, we present RoCoIn, a robust cooperative inference mechanism for locally distributed execution of deep neural network-based inference tasks over heterogeneous edge devices. It creates a set of independent and compact student models that are learned from a large model using knowledge distillation for distributed deployment. In particular, the devices are strategically grouped to redundantly deploy and execute the same student model such that the inference process is resilient to any local failures, while a joint knowledge partition and student model assignment scheme are designed to minimize the response latency of the distributed inference system in the presence of devices with diverse capacities. Extensive simulations are conducted to corroborate the superior performance of our RoCoIn for distributed inference compared to several baselines, and the results demonstrate its efficacy in timely inference and failure resiliency.

RoCoIn是一种用于异构边缘设备上本地分布式执行深度神经网络推断任务的稳健合作推断机制，通过知识蒸馏从大型模型中学习独立而紧凑的学生模型，并以冗余方式部署和执行相同的学生模型以提高推断过程的容错性和系统响应速度。与几种基线方法相比，广泛的模拟验证了我们的RoCoIn在分布式推断中的卓越性能，并证明了其在及时推断和容错能力方面的有效性。

具有模型压缩的异构边缘设备上具有容错性的分布式推理