Identifying creative harmful memes via prompt based approach

Junhui Ji, Wei Ren, Usman Naseem

Research output: Chapter in Book/Report/Conference proceedingConference proceeding contributionpeer-review

13 Citations (Scopus)

Abstract

The creative nature of memes has made it possible for harmful content to spread quickly and widely on the internet. Harmful memes can range from spreading hate speech promoting violence, and causing emotional distress to individuals or communities. These memes are often designed to be misleading, manipulative, and controversial, making it challenging to detect and remove them from online platforms. Previous studies focused on how to fuse visual and language modalities to capture contextual information. However, meme analysis still severely suffers from data deficiency, resulting in insufficient learning of fusion modules. Further, using conventional pretrained encoders for text and images exhibits a greater semantic gap in feature spaces and leads to low performance. To address these gaps, this paper reformulates a harmful meme analysis as an auto-filling and presents a prompt-based approach to identify harmful memes. Specifically, we first transform multimodal data to a single (i.e., textual) modality by generating the captions and attributes of the visual data and then prepend the textual data in the prompt-based pre-trained language model. Experimental results on two benchmark harmful memes datasets demonstrate that our method outperformed state-of-the-art methods. We conclude with the transferability and robustness of our approach to identify creative harmful memes.

Original languageEnglish
Title of host publicationThe ACM Web Conference 2023
Subtitle of host publicationproceedings of the World Wide Web Conference WWW 2023
Place of PublicationNew York
PublisherAssociation for Computing Machinery, Inc
Pages3868-3872
Number of pages5
ISBN (Electronic)9781450394161
DOIs
Publication statusPublished - 2023
Externally publishedYes
Event2023 World Wide Web Conference, WWW 2023 - Austin, United States
Duration: 30 Apr 20234 May 2023

Conference

Conference2023 World Wide Web Conference, WWW 2023
Country/TerritoryUnited States
CityAustin
Period30/04/234/05/23

Fingerprint

Dive into the research topics of 'Identifying creative harmful memes via prompt based approach'. Together they form a unique fingerprint.

Cite this