Research into adversarially hardened network intrusion detection systems suggests that improving overall AI robustness may not protect every attack class equally.
Artificial intelligence is increasingly being used to detect suspicious network activity, classify attacks and support automated cybersecurity decisions. As these systems become more capable, researchers are also working to make them more resistant to adversarial attacks designed to manipulate machine-learning models.
But a new line of research by AI and cybersecurity researcher Rumman Firdos asks a deeper question: what happens when an intrusion detection model becomes more robust overall, but that robustness is not distributed evenly across different types of network traffic and attacks?
Firdos’ research examines this problem through what the study describes as “Vulnerability Shift” — a class-level view of adversarial robustness that looks beyond aggregate security metrics to identify where a model may become substantially more or less vulnerable.
The research, titled “Mitigating Vulnerability Shifts in Adversarially Hardened Network Intrusion Detection Systems via Multi-Epsilon Curriculum Learning,” investigates whether improvements in aggregate adversarial robustness necessarily translate into comparable improvements across individual attack classes.
The central argument is straightforward: in cybersecurity, an average robustness score may hide the weakness that matters most.
An attacker does not necessarily need to defeat an entire intrusion detection system. If a particular attack category becomes significantly easier to misclassify under adversarial conditions, that class-specific weakness could become the more relevant security concern.
Looking Beyond Aggregate Robustness
Conventional evaluations of machine-learning security systems often emphasize aggregate measurements such as overall accuracy, macro-F1 or overall robustness.
Firdos’ research proposes examining robustness at the class level instead.
The study introduces a gradient-based Sensitivity Index (Ψ) to measure the adversarial sensitivity of individual classes. The reported analysis found a strong relationship between class-level sensitivity and F1-score degradation, with a Spearman correlation of ρ = 0.8833 and p = 0.00159.
The research also proposes the Defense Budget Exhaustion (DBE) hypothesis as a possible mechanism for explaining why robustness may be distributed unevenly across classes. DBE is presented as a research hypothesis and mechanism for further investigation rather than as an established universal law.
Testing the Problem Under Adversarial Attacks
The research evaluates three adversarial attack methods: Fast Gradient Sign Method (FGSM), Projected Gradient Descent (PGD) and Auto-PGD (APGD).
Experiments were conducted using the UNSW-NB15 and CIC-IDS2017 network intrusion datasets and evaluated across Multi-Layer Perceptron (MLP) and Long Short-Term Memory (LSTM) architectures.
The reported UNSW-NB15 results illustrate the difference between classes particularly clearly.
For the Reconnaissance class, the study reports F1 degradation of 89.32% under FGSM, 96.07% under PGD and 96.75% under APGD. By comparison, the reported degradation for the Normal class remained around 0.03–0.04%.
The contrast is central to the research: adversarial conditions can affect different classes very differently, even within the same AI-based intrusion detection system.
A Multi-Epsilon Approach to Mitigation
The research proposes Multi-Epsilon Adversarial Training (MEAT) as a potential mitigation strategy.
Instead of relying on a single adversarial perturbation strength, the curriculum-based approach exposes models to multiple epsilon levels with the objective of encouraging more balanced robustness across threat categories.
In the reported experiments, MEAT produced a +0.60 F1 improvement for the Reconnaissance class under PGD.
The proposed approach is intended to provide researchers and security engineers with another way to investigate and reduce class-specific weaknesses during adversarial training.
Why Class-Level AI Security Matters
The implications extend beyond a single intrusion detection experiment.
As organizations increasingly deploy machine-learning systems for intrusion detection, threat classification and automated security analysis, the question is no longer simply whether an AI model is robust. It is also whether the model remains robust across the categories that matter operationally.
A model can perform strongly on aggregate while containing a substantially weaker decision boundary for a particular class.
That does not mean that every commercial AI security product exhibits Vulnerability Shift, nor does the research establish that the phenomenon is universal. Instead, the work identifies a potential evaluation gap and proposes a framework for investigating it.
For AI security engineering, that distinction matters. Robustness should be examined not only as a single model-wide property, but also as a distribution of resilience across the threats a system is expected to detect.
From a Research Question to a Broader Direction
The work brings together four central ideas: Vulnerability Shift, the proposed Defense Budget Exhaustion hypothesis, the class-level Sensitivity Index (Ψ), and Multi-Epsilon Adversarial Training.
Together, they provide a framework for asking a different kind of question about adversarially hardened cybersecurity systems: not simply “Did the model become more robust?” but “Which classes became more robust, which remained vulnerable, and where did robustness shift?”
For Firdos, the broader objective is to make these differences measurable.
“AI security cannot be evaluated only by asking whether a model becomes more robust overall. We also need to understand where that robustness is being gained, where it is being lost, and why,” said Firdos. “Vulnerability Shift is an attempt to make that imbalance measurable and visible.”
The research contributes to ongoing work in adversarial machine learning, AI security, robust machine learning, network intrusion detection and cybersecurity engineering.
Research at a glance
Researcher: Rumman Firdos
Research area: Adversarial machine learning, AI security and network intrusion detection
Core concept: Vulnerability Shift
Proposed mechanism: Defense Budget Exhaustion (DBE) hypothesis
Sensitivity measure: Class-level Sensitivity Index (Ψ)
Mitigation approach: Multi-Epsilon Adversarial Training (MEAT)
Datasets: UNSW-NB15 and CIC-IDS2017
Architectures: MLP and LSTM
Adversarial attacks: FGSM, PGD and APGD
