Enhancing the transferability of adversarial attacks with majority-vote

Fast Gradient Sign Method (FGSM)-based ap proaches play a critical role in adversarial attacks, particularly in transfer-based attacks. Recent advancements aim to enhance adversarial transferability by refining the sign function in FGSMs using precise gradient information. However, due to the instability of gradient update directions, such precise gradient information may still converge to local minima, thereby limiting transferability. To address this issue, we propose Majority-vote Gradient Scaling (MVGS), a novel method designed to sta bilize gradient update directions. MVGS dynamically rescales momentum terms by evaluating the consistency of historical gradients at each pixel. Since MVGS operates solely through parameter-specific momentum rescaling, it can be seamlessly integrated with existing adversarial attack methods that lever age precise gradient information. Extensive comparative and ablation experiments demonstrate that MVGS significantly improves attack success rates (ASR), achieving up to a 6.1% increase over the state-of-the-art APAA method, confirming its effectiveness.