Robust foreground object segmentation from handheld camera videos with occlusion map

Hao Xiong, Zhiyong Wang*, Renjie He, Dagan Feng

*Corresponding author for this work

Research output: Contribution to journalArticlepeer-review


Extracting foreground objects from videos captured by a handheld camera has emerged as a new challenge. While existing approaches aim to exploit several clues such as depth and motion to extract the foreground layer, there are limitations in handling partial movement and cast shadow. In this paper, we bring a novel perspective to address these two issues by utilizing occlusion map introduced by object and camera motion and taking the advantage of interactive image segmentation methods. For partial movement, we treat each video frame as an image and synthesize "seeding" user interactions (i.e., user manually marking foreground and background) with both forward and backward occlusion maps to leverage the advances in high quality interactive image segmentation. For cast shadow, we utilize a paired region based shadow detection method to further refine initial segmentation results by removing detected shadow regions. Experimental results from both qualitative evaluation and quantitative evaluation on the Hopkins dataset demonstrate both the effectiveness and the efficiency of our proposed approach.

Original languageEnglish
Pages (from-to)5751-5776
Number of pages26
JournalMultimedia Tools and Applications
Issue number10
Publication statusPublished - May 2016
Externally publishedYes


  • Video object segmentation
  • Occlusion map
  • Shadow removal
  • Handheld camera

Cite this