Your training data is boring: Anomaly detection with LLMs

Following on from research presented at Black Hat USA 2025, Sophos data scientists Ben Gelman, Tamas Nyiri, and Tibor Kristóf Lányi recently showcased new work on anomaly detection at BSides Las Vegas 2026. As promised, we’re sharing more details of that research (the recording of the talk is also available here).

Supervised classifiers in cybersecurity are often trained on data that doesn't capture the most unusual examples. Benign training sets are dominated by simple, common patterns, while malicious sets reflect known attacks from past investigations. The long tail of unusual files on both sides goes unlabeled.

When these files appear in production, classifiers are more likely to misclassify them, generating false positives that overwhelm SOCs, and false negatives that let threats through.

Traditional approaches to labeling difficult examples rarely scale. Doing it manually is expensive, rule-based collection is too narrow, and anomaly detection alone has historically produced unacceptable false positive rates. But anomaly detection combined with LLMs is excellent at something else: finding and labeling the unusual data that is missing from cybersecurity training sets.

The team developed an automated pipeline that combines anomaly detection and LLMs to augment training data for a suite of cybersecurity models, using complementary anomaly detection methods, viable across feature representations, to surface distinctly unusual data. Then, an LLM classified each anomaly with format-specific prompts calibrated per data type.

Critically, Ben and his colleagues used separate prompts that err toward malicious and benign respectively, achieving high precision on both label types. More specifically, one prompt instructed the LLM to only evaluate the input as benign when it was confident that it was benign; the other prompt instructed the same but with the malicious label. This way, they only accepted the labels when the two LLMs with the two different instructions agreed, otherwise they threw that example away and didn’t use it for training. In practice, this method produced labels with better precision (tested on labeled data). The labeled anomalies augment the training data for cybersecurity classifiers.

The Sophos AI team evaluated their method across three structurally different data types with monthly ingestion scales spanning four orders of magnitude: command lines, HTML, and RTF.

The problem

Every detection team is familiar with the core issue: supervised classifiers are only as good as their training data, and that data is systematically biased. Benign training data comes from sources chosen for labeling confidence.

Known-clean networks, corporate environments without incidents and regex-capturable patterns are obvious choices, but they inherently exclude the long tail of unusual but legitimate data.

Malicious training data is similarly constrained to known attack types from incident investigations. The events that classifiers get wrong in production are precisely the unusual events missing from both sides of the training set.

Using command lines as an example: prior work on anomaly detection for command-line classification has attempted to address labeling oversights by surfacing anomalies as direct alerts. SCADE uses isolation forests and engineered features to flag potentially malicious command lines at scale, but reports false positive rates around 2%. Filar and French discover anomalous parent-child process relationships using tree-based models, reporting similar ~3% false positive rates. Nisslmueller combines explicit command-line featurization with parent-child context for unsupervised LOLBin detection, also landing under 2% FP.

Although these rates sound low, at tens of millions of daily events they still produce overwhelming alert volumes. All three share the fundamental limitation of treating anomaly detection as an end-to-end alerting mechanism.

Ben and team’s approach sidesteps this limitation entirely: anomaly detection surfaces candidates, an LLM labels them, and the labels improve a downstream supervised classifier.

A key takeaway from the team’s work is the prompt strategy for achieving high precision on both malicious and benign labels. A single prompt struggles to simultaneously maintain high precision on both labels. Instead, the team used two separate prompts per file type: one that errs on the side of calling files malicious (producing high-precision benign labels), and another that errs on the side of calling files benign (producing high-precision malicious labels). This allows both label types from anomaly detection to be usable as training data, which was not achievable in prior work that relied on a single labeling pass.

Stand back, we’re doing science

The pipeline for Ben and team’s research was as follows:

1. Raw production telemetry
2. Feature extraction
3. Multi-method anomaly detection
4. LLM classification
5. Newly labeled training data from the long tail
6. Supervised classifier improvement

A key architectural decision was to treat anomaly detection not as the detection system itself, but as a prefilter that surfaces candidates for LLM labeling, which in turn produces training data for a supervised classifier whose precision can be controlled empirically.

Because each anomaly detection method has blind spots – it catches certain types of unusual data and misses others – Sophos AI used three methods (isolation forest, k-means anomaly, and PCA reconstruction error) on different feature representations, producing broader coverage with low overlap between methods.

The team employed several different feature sets, which contributed to data diversity:

• Hand-crafted features: Catching structural anomalies, such as files with extreme length, abnormal Shannon entropy, rare character distributions. Hand-crafted features included character-level statistics, operator/case proportions, and file-type-specific markers.
• TF-IDF vectors: To catch content-level outliers, e.g., files whose token distribution didn't match any typical cluster.
• Binary vocabulary vectors: For the unusual feature correlations, e.g., files with atypical token co-occurrence patterns.

Prompt needed

A single LLM prompt can’t always achieve high precision on both malicious and benign labels simultaneously, especially in cases where the problem itself is inherently ambiguous, such as dual use scripts . For example, if the prompt says, ‘when uncertain, lean toward malicious,’ benign precision is high, but malicious precision suffers and vice versa.

Sophos AI’s solution was to run two separate prompts per file type. One erred towards malicious (producing high-precision benign labels), the other toward benign (producing high-precision malicious labels). Each prompt was calibrated to the file type's specific attack surface.

For example, with HTML, the malicious-leaning prompt defined what normal HTML looks like (JavaScript, forms, analytics) so it didn’t over-flag standard web features. The benign-leaning prompt defined specific malicious indicators (obfuscated payloads, credential phishing, exploit kit structures), so it didn’t under-flag real threats.

Putting it to the test

Ben and colleagues evaluated this method across three structurally different data types:

• Command lines: ~10,000,000 monthly ingest. Attack surface: LOLBin abuse, obfuscated PowerShell, encoded payloads.
• HTML files: ~1,000,000 monthly ingest. Attack surface: exploit kits, credential phishing, obfuscated JavaScript.
• RTF documents: ~10,000 monthly ingest (testing under data scarcity). Attack surface: Equation Editor exploits, DDE attacks, malicious OLE.

The results on LOLBins can be found in the previous research. But expanding the work to HTML and RTF files elucidated more generalizable trends, and suggests that the method can work with a variety of data types.

Table 1: HTML results

Table 2: RTF result

Like LOLBins, RTF performed best with isolation forest, showing a 2.41% increase over the existing model, which is an encouraging jump at such a high ROC AUC.

With HTML, we see a different result. The baseline performance of the model is higher, so the increase is more modest, but the team found that K-means was the most effective with a 0.68% increase over baseline. They also noted that PCA decreased the score, which highlights exactly why they used three different anomaly detection methods: they didn’t know in advance which method would capture the most valuable anomalies in a particular dataset. And since running these algorithms is relatively cheap, the small extra cost only comes from running the evaluations on the extra anomalies with the LLM.

Try it yourself

Here’s something our data science-minded readers can try out for themselves:

• Pick a file type where your classifier has a false positive or false negative problem
• Pull a sample of unlabeled production data, featurize it, and run Isolation Forest (it's 10 lines of sklearn)
• Send the top anomalies to an LLM with a format-specific prompt. Use two prompts: one that errs malicious, one that errs benign
• (Optional) Check the quality of the LLM output on a sample either by running it on a set of your labeled data or even better, have an analyst verify the unlabeled ones
• Add the high-precision labels to your next training run and measure the difference
• Wrap it in a YAML config and schedule it. All being well, you now have a self-improving data pipeline

Top takeaways

• Classifiers’ failure cases are a training data diversity problem; anomaly detection finds the unusual files that rules and manual labeling miss
• Multi-method anomaly detection (IF + KMeans + PCA) on different feature representations catches more than any single method
• Dual-prompt LLM labeling achieves high precision on both malicious and benign labels, making both usable as training data
• The pipeline is filetype-agnostic (as long as an LLM can produce high precision labels for them – test this empirically first) and runs on a single machine. A new data type therefore requires a config file, rather than having to become an onerous engineering project
• Although research on traditional ML models has slowed in recent times, this research shows that LLMs can help revitalize the field by infusing them into the process for significant performance gains