Skip to main navigation Skip to search Skip to main content

Facial expression recognition with heatmap neighbor contrastive learning

  • Tong Liu
  • , Jing Li*
  • , Jia Wu
  • , Bo Du*
  • , Yibing Zhan
  • , Dapeng Tao
  • , Jun Wan
  • *Corresponding author for this work

Research output: Contribution to journal › Article › peer-review

Abstract

Many supervised learning-based facial expression recognition (FER) methods achieve good performance with the assistance of expression labels and a complex framework. However, there are inconsistent annotations in different expression datasets, making the above methods disadvantageous for new expression datasets or datasets with limited training data. The objective of this paper is to learn self-supervised facial expression features that enable the FER model not to rely on the annotation consistency of the different datasets. Most current self-supervised learning algorithms based on contrastive learning learn the representation by forcing different augmented views of the same image close in the embedding space, but they cannot cover all variances within a semantic class. We propose a heatmap neighbor contrastive learning (HNCL) method for FER. It treats the images corresponding to the heatmap nearest neighbors of expressions as other positives, providing more semantic variations than pre-defined augmented transformations. Therefore, our HNCL can learn better expression features covering more intra-class variances, improving the performance of the FER model based on self-supervised learning. After fine-tuning, HNCL with a simple framework achieves top-three performance on the in-the-lab datasets and even matches the performance of state-of-the-art supervised learning methods on the in-the-wild datasets.

Original languageEnglish
Pages (from-to)4795-4807
Number of pages13
JournalIEEE Transactions on Multimedia
Volume27
DOIs
Publication statusPublished - 2025

Fingerprint

Dive into the research topics of 'Facial expression recognition with heatmap neighbor contrastive learning'. Together they form a unique fingerprint.

Cite this