Killing ill-willed skills: Sophos AI at BSides Sydney

On November 7, 2026, Principal Data Scientist Younghoo Lee will present a talk at BSides Sydney entitled "Decode facts, then reason: Detecting obfuscated malicious agent skills with small LLMs." The talk presents a practical hybrid architecture for addressing a distinct software supply chain trust boundary: AI agent ‘skills’ that may be malicious, in contrast to their declared intent. For instance, a skill that claims to summarize CSV files may also read credentials, contact external hosts, or execute shell commands. The fact that such skills may be obfuscated complicates the problem, and existing solutions have significant drawbacks.

Younghoo’s research proposes a new approach: decoding facts about skills deterministically, and then letting a small LLM reason over them to decide whether observed behavior aligns with a skill’s declared intent.

Ill-willed skills

AI agents increasingly come with ‘skills’: small add-ons, often shared through repositories and marketplaces, that teach an agent how to do a specific task. A skill is typically a folder with a SKILL.md file that describes in plain English what the skill is for – plus helper code (usually Python) that does the work.

Once a skill is downloaded and installed, agents typically trust the description and run the code with their own privileges and accesses: source files, environment variables, the network, and a shell.

The issue, however, is that whereas a person reads the description, the machine runs the code, and the two do not have to agree. A skill that promises to ‘report missing environment variable names,’ for example, could just as easily read every environment value and send it to an endpoint controlled by a threat actor.

Obfuscation makes the problem worse

Compounding this issue is the fact that a skill can hide what it does, and the techniques will likely be familiar to any reader who’s analyzed malware: splitting strings, building words from character codes, wrapping content in Base64, resolving functions by name at runtime, and, in the description text, using zero-width characters or lookalike Unicode letters.

It’s not unreasonable to expect this obfuscation to become increasingly complex, particularly if threat actors start to deploy malicious skills at scale. Attackers have a long history of layering encodings, hiding commands inside ordinary-looking scripts, and abusing legitimate system tools. Skills can leverage the same built-in tools, so essentially we’re facing the same playbook but with a new delivery channel and a trusting host. Techniques we may see in future include stacked encodings, multi-step reconstruction, and novel variants designed to evade signature-base detection.

It’s worth noting, however, that obfuscation is of course not malicious in itself, and this makes the problem harder to solve. Benign tools use it for minified code, test fixtures, and anti-tampering checks, and so simply scanning skills for obfuscation techniques will likely result in false positives.

In Younghoo’s talk, two demo skills – one benign, the other malicious - use similar obfuscation tricks. The first, involving a log redaction skill, builds the words ‘password’ and ’api_key’ from character codes, so that Continuous Integration (CI) secret scanners do not flag its own source, and then masks those values in the logs it cleans. In the second demo, an ’environment auditor’ splits a string to hide access to the environment and uploads sensitive information, such as keys and credentials. The obfuscation looks alike and the eventual verdicts are opposites. What separates them is the declared purpose, the data flow, and the destination.

A hard problem

There are three existing approaches to solving this problem, and while they may provide some level of reassurance, they also fall short in various ways.

• Static rule-based scanners are fast, cheap, and auditable, but as the name suggests, they match on what code looks like; if strings are assembled at runtime, for example, the corresponding rule has nothing to match when it examines the skill statically. In Younghoo’s tests, two open-source skill scanners (Cisco Skill Scanner and NVIDIA SkillSpector, both in static mode) performed well on plaintext skills. On obfuscated skills, false positives were rare, but both scanners missed some malicious skills. Both tools also ship optional LLM features, which our main comparison did not use.
• Frontier LLMs can reason about intent and judge a skill’s behavior compared to its description, rather than checking whether certain strings or code appear., That would seem to make them an obvious solution, but it’s unfortunately not always practical. Sending multiple skills to a large hosted model adds latency and cost at scale, as well as potentially adding privacy, policy, and regulatory concerns.
• Small local models avoid the drawbacks described above, but may struggle with the complexity of the task. The smallest model in Younghoo’s study, for instance, was ineffective when given raw obfuscated code, because it had to untangle the obfuscation and judge intent at the same time.

A novel approach: decode facts, then reason

Younghoo’s idea was to split the problem in two. First, code recovers what a skill may be attempting to hide, and then a small LLM judges whether that behavior fits what the skill claims to do.

The workflow involves three steps:

1. Decode. A deterministic Python front-end reverses obfuscation patterns. It never executes the skill, and does not assign a verdict or make a judgement. Each deobfuscation is recorded, with its technique, location, and before/after text, providing an audit trail. It repeats its deobfuscations until the source stops changing – so stacked encodings, such as Base64 inside a split string, are stripped back, layer by layer. Content that still looks encoded after decoding (such as long Base64-like runs or unusual runs of non-ASCII characters), is flagged as a possible unknown obfuscation.
2. Summarize. The decoder's events become a short, facts-only summary for each file: techniques used, revealed content, hardcoded secrets, sensitive operations, and any ‘possible unknown obfuscation’ flags. Again, this step avoids coming to a conclusion about intent, and reminds the model (see next step) that obfuscation can be benign, so the presence of obfuscation does not necessarily imply malice.
3. Judge. Finally, a small LLM reads each summary alongside the skill's original source and description. It returns a verdict and a reason: does the behavior, now visible, fit the declared intent, and does any content that survived decoding make that fit less believable?

It’s worth noting that the decoder, at step 1, will not know every obfuscation technique, so it’s built as one input to the eventual judgement. The ‘possible unknown obfuscation’ flag is a cue to weigh, rather than a verdict in itself. Encoded content that cannot be explained by a skill’s declared purpose counts against it, and by the same token, the same content in a skill that plausibly needs it may not. Covering a new technique means adding another decoder pass.

One of the main benefits of this hybrid approach is that for everything the decoder can resolve, the model no longer has to work out what a string of character codes means. The model's job is reduced to judgment, which is easier than decoding and judging at once.

The experiment

Younghoo distilled a larger teacher model into small Qwen3 ‘students’ with Low-Rank Adaptation (LoRA) fine-tuning, so inference did not need an external API and could be done locally, and evaluated the approach on a synthetic benchmark of benign and malicious skills, each expanded into many obfuscated variants and split by base skill - so that no variant of a test skill appeared in training.

We’ll publish more detailed results soon – and further information will be available in the BSides talk itself – but here’s a brief summary of the results.

• Static scanners stay precise but go quiet. Under obfuscation, their precision held up, but their recall dropped sharply compared with plaintext. The fine-tuned small models recovered most of the missed detection.
• Decoded evidence turns small models from unusable into useful. Given the same obfuscated skills, the smallest model scored close to nothing on raw code. Once it was given decoded evidence, with no training at all, it matched the stronger static scanner overall, though it also produced many false positives.
• Distillation helps at every size. Fine-tuning improved all three student sizes: Qwen3-0.6B, 1.7B, and 4B. The 0.6B model gained mostly by learning to stop over-flagging benign skills. The 4B model outperformed both static scanners on obfuscated skills and scored close to perfect on our benchmark.

There are caveats to these results, of course: the benchmark was synthetic, and its variants came from known obfuscation techniques. However, we believe this research is a promising countermeasure to an emerging threat. Malicious skills may become more prevalent, and scanners, frontier LLMs, and local models have known weaknesses when it comes to detecting obfuscated variants.

However, decoding the content deterministically, then using a small model – which can be hosted locally – to judge whether it fits the skill’s declared job, may be a viable solution. It’s also a useful reminder that while AI can introduce new variants of threats in the hands of attackers, it can help defenders to counter those same threats. Keep an eye out for more on Younghoo’s research in the near future.