Prompting the priorities: a first look at evaluating LLMs for vulnerability triage and prioritization

Security analysts are under growing pressure to assess increasingly complex and high-volume vulnerabilities. Recent advancements have shown that Large Language Models (LLMs) offer a promising solution by automating key aspects of this process through advanced natural language understanding.

In this study, we evaluate four state-of-the-art LLMs—ChatGPT, Claude, Gemini, and DeepSeek—across twelve prompting techniques (PT) to assess their effectiveness in interpreting semi-structured and unstructured vulnerability information. As a practical use case, we investigate each model’s ability to predict decision points within the Stakeholder-Specific Vulnerability Categorization (SSVC) framework, a widely used method for guiding response actions. Specifically, we focus on the four core SSVC decision points: Exploitation, Automatable, Technical Impact, and Mission & Wellbeing.

We leveraged a sample of 384 real-world vulnerabilities from the VulZoo dataset to evaluate the performance of each LLM across twelve PTs, including One-shot, Few-shot, and Chain-of-Thought. Over 165,000 LLM queries were issued to assess their predictive accuracy. Model responses were evaluated using F1-scores for each SSVC decision point (SDP), and weighted and unweighted Cohen’s Kappa metrics for SSVC decision outcomes (SDO). Our results show that Gemini consistently outperformed the other models, demonstrating superior performance on three out of four SDPs and yielding the highest number of correct recommendations. PTs that included exemplars generally led to improved accuracy, although all models exhibited difficulty with certain SDPs. Notably, only DeepSeek achieved fair agreement under weighted metrics, and all models showed a tendency to over-predict risk levels.

Our results suggest that while LLMs cannot yet replace expert judgment, certain LLM-PT combinations demonstrate moderate efficacy in specific decision areas in the SSVC vulnerability
framework. When used appropriately, LLMs can serve as effective aids in vulnerability prioritization workflows, helping security analysts respond more efficiently to emerging threats