[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"nav-notes-count":3,"content-note-ai-model-benchmarks-that-help-with-real-decisions":4},9,{"id":5,"title":6,"author":7,"authorAvatar":8,"authorRole":9,"body":10,"category":397,"date":398,"description":150,"excerpt":399,"extension":400,"image":401,"index":402,"meta":403,"navigation":417,"path":418,"pullQuote":419,"readingTime":420,"seo":421,"slug":422,"stem":423,"takeaways":424,"total":402,"__hash__":428},"notes\u002Fnotes\u002Fai-model-benchmarks-that-help-with-real-decisions.md","AI Model Benchmarks That Help With Real Decisions","Norbert Brett","\u002Fimages\u002Fnorbert-brett.jpg","Full-Stack & AI Systems Engineer",{"type":11,"value":12,"toc":387},"minimark",[13,18,31,52,111,118,122,125,134,144,182,185,224,234,246,257,284,288,295,380,383],[14,15,17],"h2",{"id":16},"why-public-ai-leaderboards-do-not-predict-real-world-reliability","Why Public AI Leaderboards Do Not Predict Real-World Reliability",[19,20,21,22,26,27,30],"p",{},"Whenever a major AI lab announces a new model, social media fills up within ten minutes with colorful bar charts showing a 2% lead on public academic tests like ",[23,24,25],"em",{},"MMLU"," or ",[23,28,29],{},"GPQA",". Then an engineering team swaps that new model into their live product on a Tuesday morning—and discovers that while the new model is great at PhD physics riddles, it suddenly ignores their formatting rules, breaks their data extractor, or starts ending every customer email with a preachy three-sentence life lesson.",[19,32,33,34,38,39,42,43,46,47,51],{},"If you have ever had to roll back a \"smarter\" AI model upgrade because it broke a working workflow, you have learned a practical rule: ",[35,36,37],"strong",{},"public leaderboards measure general trivia; only a test built from your own work tells you if a model is safe to deploy."," Months earlier, while building ",[35,40,41],{},"Humanizer 2.0",", I was frustrated by how unreliable \"asking another AI to grade the output from 1 to 10\" (",[23,44,45],{},"LLM-as-a-judge",") was: an AI judge happily gives its own robotic clichés a ",[48,49,50],"code",{},"9.5 \u002F 10",".",[53,54,55,60],"blockquote",{},[19,56,57],{},[35,58,59],{},"In Plain English — AI Evaluation Jargon Decoded:",[61,62,63,70,80,101],"ul",{},[64,65,66,69],"li",{},[35,67,68],{},"Public Benchmark (MMLU, SWE-bench)",": Standardized exams made of thousands of math, science, and coding puzzles used by AI companies for marketing comparisons.",[64,71,72,75,76,79],{},[35,73,74],{},"Domain Evaluation (\"Eval\")",": A small, custom checklist of ",[35,77,78],{},"25 to 50 real tasks from your own business"," (like extracting invoice totals or reviewing code diffs) that you run automatically whenever you change a prompt or test a new AI model.",[64,81,82,85,86,89,90,93,94,93,97,100],{},[35,83,84],{},"Deterministic Scorer vs. LLM-as-a-Judge",": Instead of asking an AI to subjectively \"grade\" another AI, a ",[35,87,88],{},"Deterministic Scorer"," is a regular Python or TypeScript script that checks hard facts (e.g., ",[23,91,92],{},"\"Is the file under 4,000 tokens?\"",", ",[23,95,96],{},"\"Did it touch any forbidden files?\"",[23,98,99],{},"\"Does the final paragraph contain a banned cliché phrase?\"",").",[64,102,103,106,107,110],{},[35,104,105],{},"Goodhart's Law",": The classic warning that ",[23,108,109],{},"\"when a measure becomes a target, it ceases to be a good measure\"","—meaning if you only optimize for a test score, you can pass the test while missing the real goal.",[19,112,113],{},[114,115],"img",{"alt":116,"src":117},"Repeatable Domain Evaluation Pipeline","\u002Fimages\u002Fnotes\u002Fdiagrams\u002Fai-model-benchmarks-that-help-with-real-decisions.svg",[14,119,121],{"id":120},"two-small-repeatable-evals-you-can-run-in-under-a-second","Two Small, Repeatable Evals You Can Run in Under a Second",[19,123,124],{},"You see, you do not need a research lab or a massive budget to build a benchmark that saves you hours of regressions. You just need real examples of mistakes that cost you time last month, paired with a script that checks for them automatically. Here are two evaluations that run right inside my own studio workflow:",[126,127,129,130,133],"h3",{"id":128},"_1-auditing-writing-quality-audit_narrativepy-in-humanizer-20","1. Auditing Writing Quality (",[48,131,132],{},"audit_narrative.py"," in Humanizer 2.0)",[19,135,136,137,140,141,143],{},"Based on the 2026 ",[35,138,139],{},"StoryScope \u002F NarraBench"," research paper (which measured 30 structural writing habits across 2,400 human and AI texts), I wrote a standalone Python script (",[48,142,132],{},") that audits any article in 40 milliseconds:",[145,146,151],"pre",{"className":147,"code":148,"language":149,"meta":150,"style":150},"language-bash shiki shiki-themes github-light github-dark","# Run the outside judge against any Studio Note in this portfolio\npython3 ~\u002F.gemini\u002Fconfig\u002Fskills\u002Fhumanizer\u002Fscripts\u002Faudit_narrative.py content\u002Fnotes\u002Fai-can-write-the-code-who-checks-it.md\n# Output: AI Slop Score: 0.0% (LOW RISK: DIVERSE HUMAN-LIKE NARRATIVE DISCOURSE)\n","bash","",[48,152,153,162,176],{"__ignoreMap":150},[154,155,158],"span",{"class":156,"line":157},"line",1,[154,159,161],{"class":160},"sJ8bj","# Run the outside judge against any Studio Note in this portfolio\n",[154,163,165,169,173],{"class":156,"line":164},2,[154,166,168],{"class":167},"sScJk","python3",[154,170,172],{"class":171},"sZZnC"," ~\u002F.gemini\u002Fconfig\u002Fskills\u002Fhumanizer\u002Fscripts\u002Faudit_narrative.py",[154,174,175],{"class":171}," content\u002Fnotes\u002Fai-can-write-the-code-who-checks-it.md\n",[154,177,179],{"class":156,"line":178},3,[154,180,181],{"class":160},"# Output: AI Slop Score: 0.0% (LOW RISK: DIVERSE HUMAN-LIKE NARRATIVE DISCOURSE)\n",[19,183,184],{},"Instead of guessing whether an article \"vibes human,\" the script checks concrete structural habits:",[61,186,187,201],{},[64,188,189,192,193,196,197,200],{},[35,190,191],{},"Flags AI Habits (Adds Penalty Points)",": Moralizing summary bows at the end of an article (",[23,194,195],{},"\"In the end, we realized that true engineering is...\"","), melodramatic physical clichés (",[23,198,199],{},"\"his heart hammered\"","), and repetitive cause-and-effect chains.",[64,202,203,206,207,93,210,213,214,217,218,93,221,100],{},[35,204,205],{},"Rewards Grounded Human Writing (Subtracts Slop Points)",": Specific dates and timeline jumps (",[23,208,209],{},"\"in 2025\"",[23,211,212],{},"\"months earlier\"","), direct reader conversation (",[23,215,216],{},"\"if you have ever\"","), and plain, honest reactions (",[23,219,220],{},"\"was annoyed\"",[23,222,223],{},"\"felt relieved\"",[19,225,226,227,229,230,233],{},"Every single one of the 10 articles on this blog was tested by ",[48,228,132],{}," and had to score ",[35,231,232],{},"0.0% AI Slop"," before shipping.",[126,235,237,238,241,242,245],{"id":236},"_2-auditing-code-skill-safety-scan_generated_skillpy-vitest","2. Auditing Code & Skill Safety (",[48,239,240],{},"scan_generated_skill.py"," & ",[48,243,244],{},"Vitest",")",[19,247,248,249,252,253,256],{},"In ",[35,250,251],{},"Book-to-Skill"," and ",[35,254,255],{},"Gemini Advisor",", our automated evaluation script checks three hard rules in under a second:",[258,259,260,271,278],"ol",{},[64,261,262,263,266,267,270],{},"Did the root ",[48,264,265],{},"SKILL.md"," file stay strictly under the ",[35,268,269],{},"4,000-token"," budget so it never bloats the agent's memory?",[64,272,273,274,277],{},"Did the builder stay strictly inside the allowed file list (",[48,275,276],{},"0"," unauthorized file edits)?",[64,279,280,281,283],{},"Did ",[48,282,240],{}," confirm zero prompt-injection attacks or copied copyright blocks?",[14,285,287],{"id":286},"what-the-benchmark-does-not-measure-and-why-that-list-matters-most","What the Benchmark Does NOT Measure (And Why That List Matters Most)",[19,289,290,291,294],{},"I felt relieved once these fast Python and Vitest scripts were wired into my daily workflow—they catch mechanical regressions in seconds. But the most important part of any benchmark is writing down ",[35,292,293],{},"what it cannot measure",", so humans know where to focus their attention:",[296,297,298,323],"table",{},[299,300,301],"thead",{},[302,303,304,309,316],"tr",{},[305,306,308],"th",{"align":307},"left","Our Automated Eval Script",[305,310,311,312,315],{"align":307},"What It Measures Reliably (",[48,313,314],{},"100%"," Automated)",[305,317,318,319,322],{"align":307},"What It ",[35,320,321],{},"Cannot"," Measure (Requires Human Review)",[324,325,326,342,361],"tbody",{},[302,327,328,336,339],{},[329,330,331],"td",{"align":307},[35,332,333,335],{},[48,334,132],{}," (Humanizer 2.0)",[329,337,338],{"align":307},"Structural AI writing habits: preachy endings, melodramatic clichés, flat timelines.",[329,340,341],{"align":307},"Whether the engineering advice is factually true, clear to a beginner, or useful to a client.",[302,343,344,351,358],{},[329,345,346],{"align":307},[35,347,348,349,245],{},"Boundary Reviewer (",[48,350,255],{},[329,352,353,354,357],{"align":307},"Unauthorized file edits, deleted tests, broken types (",[48,355,356],{},"94%"," pre-merge catch rate).",[329,359,360],{"align":307},"Whether the feature itself is worth building or fits the client's budget.",[302,362,363,374,377],{},[329,364,365],{"align":307},[35,366,367,370,371],{},[48,368,369],{},"vitest"," + ",[48,372,373],{},"nuxt typecheck",[329,375,376],{"align":307},"Data schemas, XSS security filters, broken image paths, and pricing math.",[329,378,379],{"align":307},"Visual harmony, photography craft, and whether the page earns a visitor's trust.",[19,381,382],{},"Build a small, fast script to police the repetitive mistakes automatically—and use the time it frees up to apply human judgment where no benchmark can reach.",[384,385,386],"style",{},"html pre.shiki code .sJ8bj, html code.shiki .sJ8bj{--shiki-default:#6A737D;--shiki-dark:#6A737D}html pre.shiki code .sScJk, html code.shiki .sScJk{--shiki-default:#6F42C1;--shiki-dark:#B392F0}html pre.shiki code .sZZnC, html code.shiki .sZZnC{--shiki-default:#032F62;--shiki-dark:#9ECBFF}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}",{"title":150,"searchDepth":164,"depth":164,"links":388},[389,390,396],{"id":16,"depth":164,"text":17},{"id":120,"depth":164,"text":121,"children":391},[392,394],{"id":128,"depth":178,"text":393},"1. Auditing Writing Quality (audit_narrative.py in Humanizer 2.0)",{"id":236,"depth":178,"text":395},"2. Auditing Code & Skill Safety (scan_generated_skill.py & Vitest)",{"id":286,"depth":164,"text":287},"Visual Philosophy","October 3, 2026",null,"md","\u002Fimages\u002Fnotes\u002Fai-model-benchmarks-that-help-with-real-decisions.webp","10",{"sections":404},[405,411],{"marker":406,"heading":407,"body":408},"\u002F A","Why Leaderboard Scores Do Not Predict Production Reliability",[409,410],"Every model launch comes with a bar chart showing a 3% lead on MMLU, GPQA, or SWE-bench. Yet if you have ever swapped a new model ID into an existing production pipeline and watched it break your JSON schema or ignore a negative constraint, you know that public benchmarks measure general capability, not task adherence.","When building Humanizer 2.0 and Gemini Advisor in 2026, I built small, deterministic Python evaluations scored by executable code rather than LLM-as-a-judge.",{"marker":412,"heading":413,"body":414},"\u002F B","Designing a Repeatable Eval—and Knowing What It Cannot Measure",[415,416],"In audit_narrative.py, we score text against concrete NarraBench discourse markers: moralizing tail paragraphs, somatic clichés, temporal anachrony, and explicit emotion labels. Every article in this feed must pass under the 35% slop threshold before shipping.","A good benchmark tells you immediately when a prompt or model change regresses on known failure modes—while leaving taste, truthfulness, and domain judgment to human review.",true,"\u002Fnotes\u002Fai-model-benchmarks-that-help-with-real-decisions","Public leaderboards tell you how well a model solves olympiad math puzzles; a 40-case domain eval tells you whether upgrading models tomorrow will break your production parser.","7 min read",{"title":6,"description":150},"ai-model-benchmarks-that-help-with-real-decisions","notes\u002Fai-model-benchmarks-that-help-with-real-decisions",[425,426,427],"Public AI leaderboards measure general trivia and math puzzles; they do not tell you if a new model will follow your business rules.","A practical internal benchmark only needs 30–50 real examples from your own work, scored by a fast Python or TypeScript script.","Always write down what your benchmark CANNOT measure (factual truth, customer empathy, product strategy) so a human still checks what matters.","BGlQCer4J0P6ALYZF8fl9yuiG0KxFDOly-gBZojmUjY"]