Purpose: Automatic segmentation and classification of surgical activity is
crucial for providing advanced support in computer-assisted interventions and
autonomous functionalities in robot-assisted surgeries. Prior works have
focused on recognizing either coarse activities, such as phases, or
fine-grained activities, such as gestures. This work aims at jointly
recognizing two complementary levels of granularity directly from videos,
namely phases and steps. Method: We introduce two correlated surgical
activities, phases and steps, for the laparoscopic gastric bypass procedure. We
propose a Multi-task Multi-Stage Temporal Convolutional Network (MTMS-TCN)
along with a multi-task Convolutional Neural Network (CNN) training setup to
jointly predict the phases and steps and benefit from their complementarity to
better evaluate the execution of the procedure. We evaluate the proposed method
on a large video dataset consisting of 40 surgical procedures (Bypass40).
Results: We present experimental results from several baseline models for both
phase and step recognition on the Bypass40 dataset. The proposed MTMS-TCN
method outperforms in both phase and step recognition by 1-2% in accuracy,
precision and recall, compared to single-task methods. Furthermore, for step
recognition, MTMS-TCN achieves a superior performance of 3-6% compared to LSTM
based models in accuracy, precision, and recall. Conclusion: In this work, we
present a multi-task multi-stage temporal convolutional network for surgical
activity recognition, which shows improved results compared to single-task
models on the Bypass40 gastric bypass dataset with multi-level annotations. The
proposed method shows that the joint modeling of phases and steps is beneficial
to improve the overall recognition of each type of activity.

本研究提出了一个多任务多阶段时间卷积网络 (MTMS-TCN)，旨在联合识别手术活动的两个互补粒度 —— 阶段和步骤，从而提高手术活动识别的精确性和召回率。在 Bypass40 数据组上进行实验表明，MTMS-TCN 在相较于单一任务方法，无论在阶段识别还是步骤识别方面，准确度、精确度和召回率上均具有 1-2% 左右的提升，并且相较于 LSTM 模型，其在步骤识别方面具有更高的性能。