Resource-constrained Edge Devices (EDs), e.g., IoT sensors and
microcontroller units, are expected to make intelligent decisions using Deep
Learning (DL) inference at the edge of the network. Toward this end, there is a
significant research effort in developing tinyML models - Deep Learning (DL)
models with reduced computation and memory storage requirements - that can be
embedded on these devices. However, tinyML models have lower inference
accuracy. On a different front, DNN partitioning and inference offloading
techniques were studied for distributed DL inference between EDs and Edge
Servers (ESs). In this paper, we explore Hierarchical Inference (HI), a novel
approach proposed by Vishnu et al. 2023, arXiv:2304.00891v1 , for performing
distributed DL inference at the edge. Under HI, for each data sample, an ED
first uses a local algorithm (e.g., a tinyML model) for inference. Depending on
the application, if the inference provided by the local algorithm is incorrect
or further assistance is required from large DL models on edge or cloud, only
then the ED offloads the data sample. At the outset, HI seems infeasible as the
ED, in general, cannot know if the local inference is sufficient or not.
Nevertheless, we present the feasibility of implementing HI for machine fault
detection and image classification applications. We demonstrate its benefits
using quantitative analysis and argue that using HI will result in low latency,
bandwidth savings, and energy savings in edge AI systems.

本文介绍了一种新的分层推理方法（hierarchical inference），旨在解决资源受限的边缘设备上深度学习推理的精度和开销问题，同时探讨了该方法在机器故障检测和图像分类应用中的可行性和优势。

网络边缘分层深度学习推断的案例

The Case for Hierarchical Deep Learning Inference at the Network Edge

We consider a resource-constrained Edge Device (ED) embedded with a
small-size ML model (S-ML) for a generic classification application, and an
Edge Server (ES) that hosts a large-size ML model (L-ML). Since the inference
accuracy of S-ML is lower than that of the L-ML, offloading all the data
samples to the ES results in high inference accuracy, but it defeats the
purpose of embedding S-ML on the ED and deprives the benefits of reduced
latency, bandwidth savings, and energy efficiency of doing local inference. To
get the best out of both worlds, i.e., the benefits of doing inference on the
ED and the benefits of doing inference on ES, we explore the idea of
Hierarchical Inference (HI), wherein S-ML inference is only accepted when it is
correct, otherwise the data sample is offloaded for L-ML inference. However,
the ideal implementation of HI is infeasible as the correctness of the S-ML
inference is not known to the ED. We thus propose an online meta-learning
framework to predict the correctness of the S-ML inference. The resulting
online learning problem turns out to be a Prediction with Expert Advice (PEA)
problem with continuous expert space. We consider the full feedback scenario,
where the ED receives feedback on the correctness of the S-ML once it accepts
the inference, and the no-local feedback scenario, where the ED does not
receive the ground truth for the classification, and propose the HIL-F and
HIL-N algorithms and prove a regret bound that is sublinear with the number of
data samples. We evaluate and benchmark the performance of the proposed
algorithms for image classification applications using four datasets, namely,
Imagenette, Imagewoof, MNIST, and CIFAR-10.

研究了嵌入小型机器学习模型的资源受限边缘设备与托管大型机器学习模型的边缘服务器之间的 Hierarchical Inference 策略，提出了在线元学习框架，以预测小型模型的分类猜测正确性，并使用四个数据集对其性能进行了评估。

深度学习应用中的分层推理在线算法

Online Algorithms for Hierarchical Inference in Deep Learning  applications at the Edge

Coalescing compact binaries emitting gravitational wave (GW) signals, as
recently detected by the Advanced LIGO-Virgo network, constitute a population
over the multi-dimensional space of component masses and spins, redshift, and
other parameters. Characterizing this population is a major goal of GW
observations and may be approached via parametric models. We demonstrate
hierarchical inference for such models with a method that accounts for
uncertainties in each binary merger's individual parameters, for mass-dependent
selection effects, and also for the presence of a second population of
candidate events caused by detector noise. Thus, the method is robust to
potential biases from a contaminated sample and allows us to extract
information from events that have a relatively small probability of
astrophysical origin.

通过分层推理，采用参数模型表征引力波信号发射的致密双星种群，考虑个体参数、质量选择效应和探测器噪音所带来的候选事件偏差，从而尽可能多地提取宇宙起源可能性相对较小的事件信息。