MaskCOV: a random mask covariance network for ultra-fine-grained visual categorization

Xiaohan Yu, Yang Zhao, Yongsheng Gao*, Shengwu Xiong

*Corresponding author for this work

Research output: Contribution to journalArticlepeer-review

38 Citations (Scopus)


Ultra-fine-grained visual categorization (ultra-FGVC) categorizes objects with more similar patterns between classes than those in fine-grained visual categorization (FGVC), e.g., where the spectrum of granularity significantly moves down from classifying species to classifying cultivars within the same species. It is considered as an open research problem mainly due to the following challenges. First, the inter-class differences among images are much smaller by level of orders (e.g., cultivars in the same species) than those in current FGVC tasks (e.g., species). Second, there is only a few samples per category, which is beyond the ability of most large training data favored convolutional neural network methods. To address these problems, we propose a novel random mask covariance network (MaskCOV), which integrates an auxiliary self-supervised learning module with a powerful in-image data augmentation scheme for the ultra-FGVC. Specifically, we first uniformly partition input images into patches and then augment data by randomly shuffling and masking these patches. On top of that, we introduce an auxiliary self-supervised learning module of predicting the spatial covariance context of these patches to increase discriminability of our network for classification. Very encouraging experimental results of the proposed method in comparison with the state-of-the-art benchmarks demonstrate its superiority and potential of MaskCOV concept, which pushes research boundary forward from the fine-grained to the ultra-fine-grained visual categorization.

Original languageEnglish
Article number108067
Pages (from-to)1-12
Number of pages12
JournalPattern Recognition
Publication statusPublished - Nov 2021
Externally publishedYes


  • Ultra-fine-grained visual categorization
  • Fine-grained visual categorization
  • Covariance matrix
  • Self-supervised learning


Dive into the research topics of 'MaskCOV: a random mask covariance network for ultra-fine-grained visual categorization'. Together they form a unique fingerprint.

Cite this