Zero of Twenty-Five Skills Were Loading

A directory one level too deep meant an entire agent skill library was invisible. Every workflow that assumed it was loaded had been running without it.

Summary

We had a library of 25 security skills for an AI coding agent, organized into a security/ subdirectory for tidiness. The agent discovers skills exactly one level deep. All 25 were invisible. Every security workflow that assumed those skills were loading had been running without them, and any before/after comparison of the library was measuring nothing.

The layout

The agent discovers skills at:

.claude/skills/<name>/SKILL.md

Exactly one level. It does not recurse into grouping subdirectories.

Our repository shipped them as:

.claude/skills/security/<name>/SKILL.md
.claude/skills/security/security-awareness/<name>/SKILL.md

One level too deep, and in some cases two.

Nothing errors. Nothing warns. The directory exists, the files are well-formed, the frontmatter is valid. The skills simply never appear.

How to actually verify it

This is the part worth keeping, because "are my skills loaded?" is otherwise unanswerable by inspection.

Run the agent in structured-output mode and read the skills array in the initialization record:

claude -p --output-format stream-json --verbose
# read the {"type":"system","subtype":"init"} record

That's the agent telling you what it actually registered. Not what's on disk, not what your config says — what loaded.

Verified against a specific agent version, which matters: discovery behavior is a product detail that can change between releases, so the probe is more durable than the finding.

Any extension mechanism that fails silently needs a "what actually loaded" probe, and you should run it before trusting any result that depends on the extensions.

Why it mattered more than it looks

Two consequences, and the second is worse than the first.

Every security workflow had been running without its skills. The agent still did the work — it's a capable model, and it produced plausible security reviews. It just wasn't using the specific guidance we'd written. Output that looks right is exactly what makes this hard to notice.

Every measurement of the library was invalid. We'd been evaluating whether the skill library improved agent behavior. It couldn't have, in either direction — both arms of every comparison were running the same unmodified agent.

That's the expensive part. A broken feature is a day of work; a broken feature that silently invalidates your evaluation of the feature can consume weeks, because every result is explainable and every conclusion is wrong.

The fix

Flatten to .claude/skills/<name>/. All 25 directories, no name collisions once flattened.

One detail worth knowing: the directory name is what gets registered, and a mismatched name: field in the frontmatter is tolerated. A directory called auth-security/ whose frontmatter declares authentication-authorization-clerk works fine. So flattening doesn't require touching file contents — just move the directories.

That's a small mercy on a 25-directory migration, and it's the kind of thing worth checking before writing a rename script.

The second problem, which flattening doesn't solve

Once the skills are discoverable, the agent still has to choose to use them.

It largely doesn't. Measured across implicit trigger phrasing ("review this for security issues") and explicit trigger phrasing (the exact words in the skill's own trigger list), the model misses both. It reliably invokes a skill only when told directly: "use the X skill."

So the honest state of things: discovery was broken, we fixed it, and the automatic routing rate is near zero anyway.

That's worth separating cleanly, because they're different problems with different owners. Discovery is a packaging bug we caused and fixed in an afternoon. Routing is a model-behavior property, and the lever for it is skill descriptions and trigger phrasing — a much longer loop.

It also means the bar for "does this skill library help?" was on the floor, which is useful context for evaluating any alternative harness. If your baseline automatic-routing rate is approximately zero, almost anything is an improvement, and you should be suspicious of how impressive that improvement sounds.

If your own rig has skills that silently refuse to load, an on-site local AI setup hands the machine back working, with notes on why.

Key takeaways

1. Extension discovery can fail silently and completely. Twenty-five skills, zero loaded, no error anywhere.
2. Probe what actually registered, not what's on disk. The agent's initialization record is authoritative; your directory listing isn't.
3. A silently broken feature invalidates every evaluation of that feature. Both arms of our comparison were the same agent, and every result was explainable.
4. Plausible output is what makes this hard to catch. The agent produced reasonable security reviews without the skills, so nothing looked wrong.
5. Check whether directory name or frontmatter name is authoritative before writing a migration. Ours registers the directory name and tolerates mismatched frontmatter.
6. Separate discovery from routing. They look like one problem ("the skills aren't being used") and have completely different fixes.