We explore the use of knowledge distillation (KD) for learning compact and
accurate models that enable classification of animal behavior from
accelerometry data on wearable devices. To this end, we take a deep and complex
convolutional neural network, known as residual neural network (ResNet), as the
teacher model. ResNet is specifically designed for multivariate time-series
classification. We use ResNet to distill the knowledge of animal behavior
classification datasets into soft labels, which consist of the predicted
pseudo-probabilities of every class for each datapoint. We then use the soft
labels to train our significantly less complex student models, which are based
on the gated recurrent unit (GRU) and multilayer perceptron (MLP). The
evaluation results using two real-world animal behavior classification datasets
show that the classification accuracy of the student GRU-MLP models improves
appreciably through KD, approaching that of the teacher ResNet model. To
further reduce the computational and memory requirements of performing
inference using the student models trained via KD, we utilize dynamic
fixed-point quantization (DQ) through an appropriate modification of the
computational graph of the considered models. We implement both unquantized and
quantized versions of the developed KD-based models on the embedded systems of
our purpose-built collar and ear tag devices to classify animal behavior in
situ and in real time. Our evaluations corroborate the effectiveness of KD and
DQ in improving the accuracy and efficiency of in-situ animal behavior
classification.

使用知识蒸馏技术，从加速度计数据中学习动物行为的紧凑而准确的模型，并利用门控循环单元（GRU）和多层感知器（MLP）作为学生模型，有效地降低了推理的计算和存储要求。进一步使用动态定点量化（DQ）技术在现场对动物行为进行分类，验证了该方法在提高准确度和效率方面的有效性。