A multimodal framework for the identification of vaccine critical memes on Twitter

Usman Naseem, Jinman Kim, Matloob Khushi, Adam G. Dunn

Research output: Chapter in Book/Report/Conference proceedingConference proceeding contributionpeer-review

4 Citations (Scopus)

Abstract

Memes can be a useful way to spread information because they are funny, easy to share, and can spread quickly and reach further than other forms. With increased interest in COVID-19 vaccines, vaccination-related memes have grown in number and reach. Memes analysis can be difficult because they use sarcasm and often require contextual understanding. Previous research has shown promising results but could be improved by capturing global and local representations within memes to model contextual information. Further, the limited public availability of annotated vaccine critical memes datasets limit our ability to design computational methods to help design targeted interventions and boost vaccine uptake. To address these gaps, we present VaxMeme, which consists of 10,244 manually labelled memes. With VaxMeme, we propose a new multimodal framework designed to improve the memes' representation by learning the global and local representations of memes. The improved memes' representations are then fed to an attentive representation learning module to capture contextual information for classification using an optimised loss function. Experimental results show that our framework outperformed state-of-the-art methods with an F1-Score of 84.2%. We further analyse the transferability and generalisability of our framework and show that understanding both modalities is important to identify vaccine critical memes on Twitter. Finally, we discuss how understanding memes can be useful in designing shareable vaccination promotion, myth debunking memes and monitoring their uptake on social media platforms.

Original languageEnglish
Title of host publicationWSDM '23
Subtitle of host publicationproceedings of the Sixteenth ACM International Conference on Web Search and Data Mining
Place of PublicationNew York
PublisherAssociation for Computing Machinery, Inc
Pages706-714
Number of pages9
Volume1
ISBN (Electronic)9781450394079
DOIs
Publication statusPublished - 2023
Externally publishedYes
Event16th ACM International Conference on Web Search and Data Mining, WSDM 2023 - Singapore, Singapore
Duration: 27 Feb 20233 Mar 2023

Conference

Conference16th ACM International Conference on Web Search and Data Mining, WSDM 2023
Country/TerritorySingapore
CitySingapore
Period27/02/233/03/23

Keywords

  • Vaccine critical memes analysis
  • Multimodal data and framework

Fingerprint

Dive into the research topics of 'A multimodal framework for the identification of vaccine critical memes on Twitter'. Together they form a unique fingerprint.

Cite this