Skip to main navigation Skip to search Skip to main content

Personalized multimodal sentiment analysis under uncertain modalities missing via pretraining and online learning

  • Hongxiang Sun
  • , Zhizhong Liu*
  • , Dianhui Chu
  • , Quan Z. Sheng
  • , Zhaowei Liu
  • , Jian Yu
  • *Corresponding author for this work

Research output: Contribution to journal › Article › peer-review

Abstract

Currently, multimodal sentiment analysis (MSA) for personalized users under uncertain modalities missing has become a new challenging problem. To address this issue, we propose a two-step idea. First, we propose an effective MSA model under uncertain modalities missing and train it with some public datasets, thus to enable the model to possess better preliminary MSA ability. Then, we make the pretrained model to continuously learn user's personalized characteristics with online learning methods, thereby enable the model grow into a robust model for personalized MSA. Based on this idea, we propose a Personalized MSA model under uncertain modalities missing via Pretraining and Online Learning (termed as PMSAPO). For Personalized MSA under uncertain modalities missing, PMSAPO firstly generates the fused modality and allocate weights for each modality with a Fully Connected Neural Network Evaluation Module. Then, PMSAPO completes the final sentiment classification based on the fusion modality with a Joint feature optimization module. For the pretrained PMSAPO, we make it autonomously learn the personalized users via our proposed online learning techniques, including an online meta-learning method, a learning rate adaptive adjustment strategy, and a dynamic weight assignment strategy for sample data. Finally, based on three public benchmark datasets (IEMOCAP, MELD and CMU-MOSI), we conduct extensive experiments and prove that PMSAPO completely outperforms the Twelve state-of-the-art baseline models. (Code is available at https://github.com/SHX-AI/PMSAPO.)

Original languageEnglish
Article number114287
Pages (from-to)1-19
Number of pages19
JournalKnowledge-Based Systems
Volume329
Issue numberPart A
DOIs
Publication statusPublished - 4 Nov 2025

Keywords

  • Multimodal sentiment analysis
  • Uncertain modalities missing
  • Pretraining
  • Online learning

Fingerprint

Dive into the research topics of 'Personalized multimodal sentiment analysis under uncertain modalities missing via pretraining and online learning'. Together they form a unique fingerprint.

Cite this