RSFomer: time series transformer for robust sports action recognition

Human activity recognition (HAR) is an evolving technique that offers innovative solutions across various domains, such as healthcare, sports training, and human-computer interactions. This paper addresses the novel challenge of video-based activity recognition, focusing on detecting and classifying athletes' actions to enable precision sports training. Conventional HAR methods based on direct video analysis incur excessive computational overhead and constrained applicability. In contrast, our novel transformer-based framework, namely RSFomer, converts videos into multivariate time series, and then detects and classifies the athletes' actions. However, sports videos often suffer from severe occlusion, which introduces significant noise to the converted time series and thus deteriorates recognition performance. To address this challenge, we implement several innovative strategies to improve the robustness of our framework. First, we propose a dual-scale filtering mechanism that leverages the unscented Kalman filter and kinematic constraints to reduce noise and outliers in the converted time series. Second, we incorporate the masking mechanism and temporal slicing mechanism to enhance the transformer's ability to handle anomalies and extract multi-scale features for accurate action recognition. We perform extensive evaluations on our Boxing dataset as well as the UEA and FineGym datasets. The results demonstrate that our RSFomer is effective, outperforming existing state-of-the-art methods with significant advantages.