Graph sequence recurrent neural network for vision-based freezing of gait detection

Kun Hu*, Zhiyong Wang, Wei Wang, Kaylena A. Ehgoetz Martens, Liang Wang, Tieniu Tan, Simon J. G. Lewis, David Dagan Feng

*Corresponding author for this work

Research output: Contribution to journalArticlepeer-review

52 Citations (Scopus)

Abstract

Freezing of gait (FoG) is one of the most common symptoms of Parkinson's disease (PD), a neurodegenerative disorder which impacts millions of people around the world. Accurate assessment of FoG is critical for the management of PD and to evaluate the efficacy of treatments. Currently, the assessment of FoG requires well-trained experts to perform time-consuming annotations via vision-based observations. Thus, automatic FoG detection algorithms are needed. In this study, we formulate vision-based FoG detection, as a fine-grained graph sequence modelling task, by representing the anatomic joints in each temporal segment with a directed graph, since FoG events can be observed through the motion patterns of joints. A novel deep learning method is proposed, namely graph sequence recurrent neural network (GS-RNN), to characterize the FoG patterns by devising graph recurrent cells, which take graph sequences of dynamic structures as inputs. For the cases of which prior edge annotations are not available, a data-driven based adjacency estimation method is further proposed. To the best of our knowledge, this is one of the first studies on vision-based FoG detection using deep neural networks designed for graph sequences of dynamic structures. Experimental results on more than 150 videos collected from 45 patients demonstrated promising performance of the proposed GS-RNN for FoG detection with an AUC value of 0.90.

Original languageEnglish
Article number8870255
Pages (from-to)1890-1901
Number of pages12
JournalIEEE Transactions on Image Processing
Volume29
Early online date15 Oct 2019
DOIs
Publication statusPublished - 2020
Externally publishedYes

Keywords

  • deep learning
  • freezing of gait detection
  • graph sequence
  • Parkinson's disease
  • recurrent neural network

Fingerprint

Dive into the research topics of 'Graph sequence recurrent neural network for vision-based freezing of gait detection'. Together they form a unique fingerprint.

Cite this