Cascade multi-level transformer network for surgical workflow analysis

Wenxi Yue, Hongen Liao, Yong Xia, Vincent Lam, Jiebo Luo, Zhiyong Wang

Research output: Contribution to journalArticlepeer-review

3 Citations (Scopus)

Abstract

Surgical workflow analysis aims to recognise surgical phases from untrimmed surgical videos. It is an integral component for enabling context-aware computer-aided surgical operating systems. Many deep learning-based methods have been developed for this task. However, most existing works aggregate homogeneous temporal context for all frames at a single level and neglect the fact that each frame has its specific need for information at multiple levels for accurate phase prediction. To fill this gap, in this paper we propose Cascade Multi-Level Transformer Network (CMTNet) composed of cascaded Adaptive Multi-Level Context Aggregation (AMCA) modules. Each AMCA module first extracts temporal context at the frame level and the phase level and then fuses frame-specific spatial feature, frame-level temporal context, and phase-level temporal context for each frame adaptively. By cascading multiple AMCA modules, CMTNet is able to gradually enrich the representation of each frame with the multi-level semantics that it specifically requires, achieving better phase prediction in a frame-adaptive manner. In addition, we propose a novel refinement loss for CMTNet, which explicitly guides each AMCA module to focus on extracting the key context for refining the prediction of the previous stage in terms of both prediction confidence and smoothness. This further enhances the quality of the extracted context effectively. Extensive experiments on the Cholec80 and the M2CAI datasets demonstrate that CMTNet achieves state-of-the-art performance.

Original languageEnglish
Pages (from-to)2817-2831
Number of pages15
JournalIEEE Transactions on Medical Imaging
Volume42
Issue number10
Early online date10 Apr 2023
DOIs
Publication statusPublished - 1 Oct 2023

Keywords

  • Feature extraction
  • Gallbladder
  • Logic gates
  • Surgery
  • Surgical Phase Recognition
  • Surgical Workflow Analysis
  • Task analysis
  • Temporal Context Aggregation
  • Transformer Network
  • Transformers
  • Videos

Fingerprint

Dive into the research topics of 'Cascade multi-level transformer network for surgical workflow analysis'. Together they form a unique fingerprint.

Cite this