Shortcut Dependence and Structural Invariance in Cross-Domain URL Classification
Working title, simplified until fully published. The description on this page accurately reflects the actual research, without disclosing implementation-level methodology.
This study tests whether a URL classifier that performs well across domains might still be quietly relying on a superficial cue rather than genuine malicious signal. Using a controlled test that changes only a URL's HTTP/HTTPS scheme while leaving everything else unchanged, the work finds that a substantial share of predictions flip class purely from this single, superficial change. Only directly removing the scheme information from the model's input eliminated the sensitivity entirely, but at a measurable cost to overall performance.
A classifier can perform well on held-out data while actually relying on cues that were never intended to be the basis for its decisions, a pattern known as shortcut learning. This matters for malicious URL detection because different datasets are collected and labeled differently, and a model can end up exploiting those incidental differences rather than genuine indicators of malicious intent.
The study uses a controlled counterfactual test: constructing pairs of URLs that are identical except for their scheme (http vs. https), then measuring how often a model's prediction flips purely because of that swap, a change that carries no inherent malicious signal on its own. Several candidate fixes were tested, including a domain-adaptation technique designed to discourage dataset-specific shortcuts, and a structural approach that removes scheme information from the input entirely before the model ever sees it.
Even a model that performed well under standard domain-disjoint evaluation showed a striking proportion of predictions flip when only the scheme was swapped. The domain-adaptation approach made little difference to this sensitivity, showing that reducing broad dataset dependence does not automatically remove reliance on one specific, superficial factor. Only structurally deleting the scheme from the input entirely eliminated the sensitivity, but at a measurable cost to overall predictive performance, confirming a genuine trade-off between accuracy and robustness to this kind of shortcut.
The broader implication runs through this whole research direction: strong aggregate cross-domain performance numbers do not guarantee a model is making decisions the way it appears to be. Targeted behavioral tests can reveal dependencies that standard accuracy or F1 metrics alone would never surface.
Full methodology, results, and the underlying paper are not disclosed here, as this research is intended for conference submission and/or currently under review.