TY - UNPB
T1 - Can Argus Judge Them All?
T2 - Comparing VLMs across domains
AU - Joshi, Harsh
AU - Kashyap, Gautam Siddharth
AU - Ali, Rafiq
AU - Shabbir, Ebad
AU - Jain, Niharika
AU - Jain, Sarthak
AU - Gao, Jiechao
AU - Naseem, Usman
PY - 2025/6/23
Y1 - 2025/6/23
N2 - Vision-Language Models (VLMs) are advancing multimodal AI, yet their performance consistency across tasks is underexamined. We benchmark CLIP, BLIP, and LXMERT across diverse datasets spanning retrieval, captioning, and reasoning. Our evaluation includes task accuracy, generation quality, efficiency, and a novel Cross-Dataset Consistency (CDC) metric. CLIP shows strongest generalization (CDC: 0.92), BLIP excels on curated data, and LXMERT leads in structured reasoning. These results expose trade-offs between generalization and specialization, informing industrial deployment of VLMs and guiding development toward robust, task-flexible architectures.
AB - Vision-Language Models (VLMs) are advancing multimodal AI, yet their performance consistency across tasks is underexamined. We benchmark CLIP, BLIP, and LXMERT across diverse datasets spanning retrieval, captioning, and reasoning. Our evaluation includes task accuracy, generation quality, efficiency, and a novel Cross-Dataset Consistency (CDC) metric. CLIP shows strongest generalization (CDC: 0.92), BLIP excels on curated data, and LXMERT leads in structured reasoning. These results expose trade-offs between generalization and specialization, informing industrial deployment of VLMs and guiding development toward robust, task-flexible architectures.
U2 - 10.48550/arXiv.2507.01042
DO - 10.48550/arXiv.2507.01042
M3 - Preprint
T3 - arXiv
BT - Can Argus Judge Them All?
ER -