Action recognition is a prerequisite for many applications in laparoscopic video analysis including but not limited to surgical training, operation room planning, follow-up surgery preparation, post-operative surgical assessment, and surgical outcome estimation. However, automatic action recognition in laparoscopic surgeries involves numerous challenges such as (I) cross-action and intra-action duration variation, (II) relevant content distortion due to smoke, blood accumulation, fast camera motions, organ movements, object occlusion, and (III) surgical scene variations due to different illuminations and viewpoints. Besides, action annotations in laparoscopy surgeries are limited and expensive due to requiring expert knowledge. In this study, we design and evaluate a CNN-RNN architecture as well as a customized training-inference framework to deal with the mentioned challenges in laparoscopic surgery action recognition. Using stacked recurrent layers, our proposed network takes advantage of inter-frame dependencies to negate the negative effect of content distortion and variation in action recognition. Furthermore, our proposed frame sampling strategy effectively manages the duration variations in surgical actions to enable action recognition with high temporal resolution. Our extensive experiments confirm the superiority of our proposed method in action recognition compared to static CNNs.

在这项研究中，我们设计并评估了一个CNN-RNN架构以及一个定制的训练-推理框架，以应对腹腔镜手术行动识别中的挑战，包括动作持续时间变异、烟雾、血液积聚、快速相机运动、器官移动、物体阻挡等相关内容扭曲，以及不同照明和视角引起的手术场景变化。通过使用堆叠的循环层，我们的网络利用帧间依赖性来消除内容扭曲和动作识别中的变异的负面影响。此外，我们提出的帧采样策略有效地管理手术动作的持续时间变化，实现了高时间分辨率的动作识别。我们广泛的实验证实了我们的方法在动作识别方面相比静态CNN更为优越。

腔镜手术视频中的动作识别