Prompt engineering adversarial attack against image captioning models

This work presents a highly effective strategy for attacking image captioning models through the use of prompt engineering. The objective of this approach is to deliberately guiding the output of LLMs and introduce dynamic noise into the original clean image captions, causing them to be categorized as a different class. Consequently, when the image captioning model is fine-tuned using adversarial captions, it will deteriorate and produce inaccurate captions for clean photos. The novelty of this attack is that it does not require the attacker to perform any model training and only require to prompt the LLMs to generate only a small amount of captions for the attack to be effective. Comprehensive experiments using GPT-3.5 indicate that with only 100 captions created by LLMs with malicious intent can significantly worsen picture captioning model performance by up to over 50% in BLEU metric and over 25% in ROUGE-L and METEOR scores.