This is the second of a three-part series.
Every organization runs a quality system that appears on no org chart.
It works the same way everywhere. Before a document goes upstairs, it routes through one particular reviewer — the trusted editor, the architect whose look-over everyone requests, the person who reads a page addressed to a VP and says, “This is framed at your altitude, not the reader’s.” No process mandates the detour. Everyone takes it anyway, because the work comes back better: the buried lede caught in ninety seconds, the hedging phrases struck, the meeting changed because the rewrite happened.
Here’s what took me embarrassingly long to see: those reviewers are not doing the work. They are judging the work. Those are different capabilities, they concentrate in different people, and organizations systematically invest in the first while leaving the second completely informal — a hallway asset, undocumented, unscalable, and gone the day the reviewer takes another job. Ask where any given quality bar is actually written down, and the honest answer in most organizations is: in somebody’s head.
When we started building our AI-native operating framework — the skills that produce drafts, the maturity ladder they climb — my first instinct with quality was to bake that judgment into every producing skill. Stuff the executive-writing standard into the proposal skill’s prompt. Stuff the security standard into the architecture skill’s prompt. Copy, paste, tweak, repeat.
That instinct turned out to be wrong, and the correction became the most consequential architectural decision in the whole system: the judges are not part of any skill. The judges are first-class components — siblings to the skills, not children of them. In the last piece I argued that autonomy and quality are two separate axes. This piece is about the machinery behind the y-axis, and why I’ve come to believe the library of evaluators will end up being worth more than the library of skills it governs.
A league doesn’t let teams hire their own referees
The analogy that organized my thinking is the referee pool.
A sports league maintains its referees as a shared, independent body. Referees don’t work for a team. They’re trained to one rulebook, assigned to whichever game needs them, and evaluated on the quality of their officiating — separately from the win-loss records of anyone they officiate. The integrity of the entire league rests on this structural fact. The moment a referee reports to a team, you no longer have a sport; you have an exhibition.
Now look at how most AI systems handle quality. Every producing prompt carries its own private officiating — a paragraph of “be accurate, be concise, follow the style guide” buried in the instructions. That’s each team hiring its own referee. The producer grades its own output, against a standard nobody else can see, in a copy that drifts out of sync with every other copy. Ten skills, ten slightly different embedded referees, zero governance.
The alternative is to run the referee pool: one library of evaluators, independently owned, independently versioned, assigned to whatever work needs judging. The same security evaluator officiates an architecture review today and a proposal tomorrow. Written once, governed once, improved once — and every skill in the system inherits the improvement the same afternoon.
Why independent judges work
This isn’t a novel idea; it’s a proven organizational pattern finally becoming cheap enough to automate.
Amazon has run a version of it for decades in hiring: the Bar Raiser, an interviewer deliberately drawn from outside the hiring team, with veto power the hiring manager cannot override. The design insight is structural independence — the person under pressure to fill the seat is, by construction, not the person who decides whether the candidate clears the bar. Nobody gets to wave their own work through the gate.
Adam Grant gives the pattern a name in Think Again: the challenge network — the people whose job is not to do your work but to poke holes in it, and whose value depends precisely on their standing apart from it. Grant’s observation is that great re-thinkers curate their challengers deliberately, the way most of us only curate our supporters.
What AI changes is the economics. The Bar Raiser program works because Amazon pays a real cost — trained humans, hours per hire — and pays it only where the stakes justify it. A written rubric plus a capable model collapses that cost. The gate that used to be affordable only for hiring decisions is now affordable for every draft, every demo, every document heading upstairs. The pattern was always right. It just got cheap.
Three moves to build the judgment library
1. Extract the judges you already have.
Your organization already employs its best judges — their standards are your most valuable unwritten IP. Sit with them and write the rubric down: what are the five to seven things checked in the first two minutes? What’s an automatic fail? What does “ready for a CEO” mean, concretely, line by line?
This is where deep expertise gets encoded and the floor rises for everyone at once. A day-one hire doesn’t carry the executive-writing bar in their head — but the executive-writing evaluator does, and the hire’s first draft gets expert-grade critique before any human reviews it. Senior taste stops being a bottleneck and becomes infrastructure.
2. Write evaluators as components, not paragraphs.
An evaluator deserves the same engineering dignity as the skills it judges: its own file, its own owner, its own version history, reviewable and diffable like any other component. Our format is markdown-first — a persona is mostly a rich rubric, and markdown carries that cleanly — with an executable escape hatch for checks that need determinism rather than judgment. Here is the reference template, the pattern every evaluator in our library copies:
---
name: executive-writing
type: evaluation # first-class type — a sibling to skills
applies_to: [internal, external]
audience_tier: [director, vp, ceo] # convened when output targets these
pass_bar: 8.0 # min weighted score to clear the gate
executable_hook: weasel-word-linter.py # optional deterministic pre-check
owner: comms-standards # a person owns this bar
---
## What this evaluates
Whether a written artifact meets the rigor required for a senior-executive
audience before it is surfaced to a human for approval.
## Rubric (weighted)
- Bottom line up front — the ask lands in the first two lines (25%)
- Data-driven — every claim carries a number or a source (25%)
- Concision — no weasel words, no run-ons, no hedging (20%)
- Structure — scannable, logically sequenced, no burial of the point (15%)
- Audience calibration — framed for the reader's altitude, not the (15%)
author's
## Trigger logic (how the harness knows to convene this judge)
Convene when:
producing_skill in {narrative-writing, proposal, meeting-prep}
AND output.audience_tier intersects [director, vp, ceo]
## Behavior
1. Run executable_hook as a deterministic pre-pass.
2. Score the draft against the rubric.
3. If score < pass_bar, return specific line-level revisions to the skill.
4. Re-score after revision; surface to a human only once the bar clears.Notice the two-layer design. The rubric is judgment — it needs a model. The hook is arithmetic — does this doc contain weasel words, does this architecture leave a port open — and deserves a deterministic check, not a judgment call. Most quality systems fail by using one layer for both jobs.
3. Convene panels by context, not by habit.
The referee pool’s real power shows up in assignment. The same architecture skill pulls security, cost, resiliency, and a red-team evaluator when the output is a workload review — and pulls the executive-communication evaluator instead when the output is the summary heading to a VP. The panel is selected by what the work is and who it’s for, at runtime, by the orchestration layer — never hard-coded into the producer.
Then comes the discipline that makes the whole thing worth building: the skill revises against the panel’s critique before the output surfaces to a human. The person approving sees only work that already cleared the bar. Review time stops being spent catching weak drafts and starts being spent on the calls only a human can make.
Who judges the judges
The obvious objection, and the one I sit with most: this could sprawl. Fifty evaluators of varying quality is just the prompt mess relocated. Our answer so far is that an evaluator is governed exactly like a skill — it declares what it scores, carries a named owner, and passes a review gate before joining the library. Judges get judged on the way in.
The subtler risk is fossilization, and I don’t think I’ve solved it. An expert’s taste evolves; a rubric holds still until someone revisits it. Write the bar down and you gain scale and lose some of the living judgment that made the bar great. My current bet is that a named owner per evaluator plus a periodic re-calibration against fresh exemplars keeps the rubrics honest — but that’s a bet, not a result, and I want to hear from anyone who has run written rubrics at scale for years rather than quarters.
The moat is the bar
Step back far enough and the produce/judge split resolves into a prediction about value.
Producing skills have a lifecycle — created, evolved, and eventually commoditized as the underlying models improve month over month; a prompt that drafts a decent proposal is close to table stakes already. What doesn’t commoditize is your organization’s encoded standard — the definition of what excellent looks like, extracted from your best people, written with enough precision that a machine can hold every draft to it at three in the morning. Two libraries, one standard: the skills produce, the evaluators judge. The league isn’t memorable because of who played. It’s memorable because the game was called straight.
Next in this series: the contract underneath both libraries — what a component must declare about itself before it’s allowed in at all, and why that contract, not the components, is what keeps fifteen skills from being fifteen clever prompts.
Until then, one exercise. List every kind of work that left your organization this quarter for a customer or an executive — proposals, architectures, escalations, the board summary. For each one, answer a single question: is the quality bar it cleared written down anywhere, or does it live in one reviewer’s head? Count the heads. Every bar that exists only in a head is an asset with a resignation date attached. If most of your answers are heads, you don’t have a quality system. You have people — and people leave.
