1. The judge must never see the source
If the judge has read what the output was supposed to say, it will confirm that the output makes sense — because it does, to someone who already knows. That is exactly the bias we have when we review our own work.
So the critic only ever sees the finished output, the way a stranger would encounter it. That one design decision matters more than everything else.
2. Ask for a retelling, not a score
“Six out of ten” is not actionable. The judge instead retells what it understood. Comparing the retelling with what the piece was meant to convey shows where comprehension broke — which scene, which transition, which unexplained object. A score says the output is bad. A retelling says why, and points at the line to change.
3. Test the test: positive controls and self-agreement
Known-good and known-bad material goes through the judge first, to check it can tell them apart. Without that, an evaluation harness is a random number generator with a confident tone.
I also measured whether the judge agrees with itself: two runs correlate at r = 0.78, with a mean absolute difference of 0.27. Only then did I trust what it said.
4. The instrument earns its cost when it contradicts you
Four times it overruled me:
- Thumbnails lie. I reported that a character stayed consistent across generated frames. At full resolution there were three different faces for one character (27/100).
- My favourite architecture lost. A “cleaner” approach I had just recommended scored 40.0 against 48.7 for the incumbent. I dropped it the same day.
- A fix made things worse. A change to correct unnatural poses dropped the image score from 36 to 28. I reverted the harmful half.
- Three data points tell any story. A trend that held at n = 3 broke at n = 5.
5. The worst bug is a missing dimension
One piece scored well but read to a human like a dream: every part intelligible, nothing connected. The harness had no axis for coherence. I added one, weighted it heavily and re-ran everything — and strong-looking scores dropped, which was correct.
An evaluation harness is software with bugs. The worst one never throws an error. It reports a good score for work that fails in a way you forgot to measure.
The numbers
| Measure | Before | After |
|---|---|---|
| Comprehension (blind judge, /10) | 3.3 | 7.8 |
| Overall (mean, /140) | — | 103.8 |
| Judge self-agreement (two runs) | — | r = 0.78 |
| Judge drift (mean abs. difference) | — | 0.27 |
| Documented test cycles | 56 |
Where this applies
- Regression tests for LLM output before a release
- A/B evaluation of prompt and model changes
- Catching quality drift after a model upgrade
- Review that scales beyond what people can read — and runs locally when the content may not leave the house
I use the same principle in production: in my meeting pipeline, every AI proposal is reviewed by a human before anything is created, and model choices are made per task after testing on real meetings. Read that case study →