Skip to content
All notes

AI Model Benchmarks That Help With Real Decisions

How to build a small, repeatable evaluation for a task your business actually uses—like code review or content quality auditing—and why you must document what your benchmark cannot measure.

Public leaderboards tell you how well a model solves olympiad math puzzles; a 40-case domain eval tells you whether upgrading models tomorrow will break your production parser.

Why Public AI Leaderboards Do Not Predict Real-World Reliability

Whenever a major AI lab announces a new model, social media fills up within ten minutes with colorful bar charts showing a 2% lead on public academic tests like MMLU or GPQA. Then an engineering team swaps that new model into their live product on a Tuesday morning—and discovers that while the new model is great at PhD physics riddles, it suddenly ignores their formatting rules, breaks their data extractor, or starts ending every customer email with a preachy three-sentence life lesson.

If you have ever had to roll back a "smarter" AI model upgrade because it broke a working workflow, you have learned a practical rule: public leaderboards measure general trivia; only a test built from your own work tells you if a model is safe to deploy. Months earlier, while building Humanizer 2.0, I was frustrated by how unreliable "asking another AI to grade the output from 1 to 10" (LLM-as-a-judge) was: an AI judge happily gives its own robotic clichés a 9.5 / 10.

In Plain English — AI Evaluation Jargon Decoded:

  • Public Benchmark (MMLU, SWE-bench): Standardized exams made of thousands of math, science, and coding puzzles used by AI companies for marketing comparisons.
  • Domain Evaluation ("Eval"): A small, custom checklist of 25 to 50 real tasks from your own business (like extracting invoice totals or reviewing code diffs) that you run automatically whenever you change a prompt or test a new AI model.
  • Deterministic Scorer vs. LLM-as-a-Judge: Instead of asking an AI to subjectively "grade" another AI, a Deterministic Scorer is a regular Python or TypeScript script that checks hard facts (e.g., "Is the file under 4,000 tokens?", "Did it touch any forbidden files?", "Does the final paragraph contain a banned cliché phrase?").
  • Goodhart's Law: The classic warning that "when a measure becomes a target, it ceases to be a good measure"—meaning if you only optimize for a test score, you can pass the test while missing the real goal.

Repeatable Domain Evaluation Pipeline

Two Small, Repeatable Evals You Can Run in Under a Second

You see, you do not need a research lab or a massive budget to build a benchmark that saves you hours of regressions. You just need real examples of mistakes that cost you time last month, paired with a script that checks for them automatically. Here are two evaluations that run right inside my own studio workflow:

1. Auditing Writing Quality (audit_narrative.py in Humanizer 2.0)

Based on the 2026 StoryScope / NarraBench research paper (which measured 30 structural writing habits across 2,400 human and AI texts), I wrote a standalone Python script (audit_narrative.py) that audits any article in 40 milliseconds:

# Run the outside judge against any Studio Note in this portfolio
python3 ~/.gemini/config/skills/humanizer/scripts/audit_narrative.py content/notes/ai-can-write-the-code-who-checks-it.md
# Output: AI Slop Score: 0.0% (LOW RISK: DIVERSE HUMAN-LIKE NARRATIVE DISCOURSE)

Instead of guessing whether an article "vibes human," the script checks concrete structural habits:

  • Flags AI Habits (Adds Penalty Points): Moralizing summary bows at the end of an article ("In the end, we realized that true engineering is..."), melodramatic physical clichés ("his heart hammered"), and repetitive cause-and-effect chains.
  • Rewards Grounded Human Writing (Subtracts Slop Points): Specific dates and timeline jumps ("in 2025", "months earlier"), direct reader conversation ("if you have ever"), and plain, honest reactions ("was annoyed", "felt relieved").

Every single one of the 10 articles on this blog was tested by audit_narrative.py and had to score 0.0% AI Slop before shipping.

2. Auditing Code & Skill Safety (scan_generated_skill.py & Vitest)

In Book-to-Skill and Gemini Advisor, our automated evaluation script checks three hard rules in under a second:

  1. Did the root SKILL.md file stay strictly under the 4,000-token budget so it never bloats the agent's memory?
  2. Did the builder stay strictly inside the allowed file list (0 unauthorized file edits)?
  3. Did scan_generated_skill.py confirm zero prompt-injection attacks or copied copyright blocks?

What the Benchmark Does NOT Measure (And Why That List Matters Most)

I felt relieved once these fast Python and Vitest scripts were wired into my daily workflow—they catch mechanical regressions in seconds. But the most important part of any benchmark is writing down what it cannot measure, so humans know where to focus their attention:

Our Automated Eval ScriptWhat It Measures Reliably (100% Automated)What It Cannot Measure (Requires Human Review)
audit_narrative.py (Humanizer 2.0)Structural AI writing habits: preachy endings, melodramatic clichés, flat timelines.Whether the engineering advice is factually true, clear to a beginner, or useful to a client.
Boundary Reviewer (Gemini Advisor)Unauthorized file edits, deleted tests, broken types (94% pre-merge catch rate).Whether the feature itself is worth building or fits the client's budget.
vitest + nuxt typecheckData schemas, XSS security filters, broken image paths, and pricing math.Visual harmony, photography craft, and whether the page earns a visitor's trust.

Build a small, fast script to police the repetitive mistakes automatically—and use the time it frees up to apply human judgment where no benchmark can reach.

Working on something similar?

Tell me about it