Most AI coding workflows — the skills, rules, and prompts people bolt onto Claude Code and Cursor — ship on vibes. Someone writes a prompt, it feels better, it gets committed. DirectiveForge treats those workflows as artifacts you can measure: it generates them, then holds them to a pre-registered eval harness that reports whether a skill actually fires when it should, and whether the model judging the output can be trusted.

DirectiveForge is an open-source (MIT) generator for AI-assisted development workflows. It produces workflow kits for Claude Code and Cursor and installs them as a one-command plugin. The interesting part isn't the generator — it's the measurement layer wrapped around it.

• 22 measured skills across three families — 12 decision skills, 6 naming skills, 4 design skills
• A pre-registered eval harness — the measurement spec is committed to git before results are collected, so metrics can't be fished for after the fact (the commit order is git-verifiable)
• One-command plugin install for Claude Code / Cursor
• Public repository and site: github.com/directiveforge/directiveforge · directiveforge.com (v0.20.0)

The harness measures two things a workflow has to get right: triggering (does the correct skill fire on the inputs it should, and stay quiet on the ones it shouldn't?) and activation repeatability (run the same case thirty times — how often does it behave the same way?).

Both are reported as Wilson 95% confidence intervals over n = 30 runs rather than single point estimates, so a change is only called an improvement when the intervals support it. Trigger quality is scored as an F1 (precision and recall together), which keeps a skill from "improving" simply by firing on everything.

Because a large part of grading is done by an LLM-as-judge, the judge is validated before its verdicts count. Against a human-scored answer key it landed 18/20 exact matches and 20/20 within ±1 — calibrated enough to trust for the deltas below.

The workflow was graded, the failing signals were fixed, and everything was re-measured against the same pre-registered spec. The published before/after:

The non-overlapping confidence intervals on activation are the point: the jump from 0.33 to 0.83 is a measured effect, not a hopeful eyeball.

One change made trigger F1 worse — it fell from 0.9091 to 0.8889. That number is in the public record on purpose. The harness caught it, and because the spec was pre-registered, the honest move was to publish the loss rather than quietly re-baseline around it. A workflow you can only ever report wins for is a workflow you aren't really measuring.

The whole system is dogfooded: DirectiveForge builds and maintains Avagyans, the production jewelry storefront I run myself, which ships a versioned kit manifest generated by the tool.

Measuring a thing honestly means publishing where it currently falls short, not only where it improved.

• v0.20.1 has not shipped. An external field trial filed 11 defects; they were dispositioned into a v0.20.1 that does not exist yet. plugin.json still pins 0.20.0.
• Two of those defects are open, and both are safety findings. D1 (blocker): the generated .mcp.json auto-executes a third-party MCP server. D2 (major): the generated settings.json widens the command allowlist. A proposals-only path exists but only fires when the target project already has its own credential-governance gate.
• One external user. Every adoption number here is n=1 outside my own projects. The harness measures the workflow, not its popularity.
• No CI, no test framework. Verification is three hand-written self-test suites (68 assertions, all passing) plus the harness itself. A generator that writes quality gates for other people's repositories does not yet run them on its own.
• The pre-registration margin is 56 minutes, not weeks. The spec commit does precede the first results commit, and that ordering is git-verifiable — but it is a same-day margin, and the delta run's prediction plan lived outside the repository, so it is operator-attested rather than git-provable. Both are disclosed in the repo's own DELTA.md.

None of these are fixed by the numbers above, and none of them are hidden in the repository either.

• Pre-registration kills metric-fishing. Committing the eval spec before collecting results — with the commit order git-verifiable — removes the temptation to redefine "better" after seeing the numbers. It's the single highest-trust decision in the project.
• Confidence intervals change what you're allowed to claim. Reporting Wilson CIs over 30 runs instead of a point estimate means "it improved" has to survive the intervals not overlapping. Most workflow tweaks don't clear that bar; the ones that do are real.
• An LLM judge is worthless until it's calibrated. Scoring the judge against a human key (18/20 exact) before trusting its verdicts is what makes LLM-as-judge defensible rather than circular.
• Publishing your own regressions is the proof. Anyone can show a green dashboard. Disclosing the F1 drop is what demonstrates the measurement is honest.