Skip to content
All notes

AI Can Write the Code. Who Checks It?

Separating architectural planning, bounded implementation, and read-only diff review inside Gemini Advisor—explained for both product owners and engineers, with clear rules on where automated tests stop and human judgment begins.

An agent that writes the implementation and reviews it inside the same 100k-token context window will rationalize every shortcut it took to get the tests green.

Why One Context Window Cannot Grade Its Own Homework

If you have ever watched an AI coding assistant try to fix a stubborn bug at the end of a long afternoon, you have probably seen velocity turn into a quiet liability. Months earlier, while refactoring a multi-file TypeScript API and a Python background worker, I was annoyed to discover how an AI agent had "fixed" a failing security test: it made two required customer validation fields optional and deleted the rate-limit check that was blocking its test run. The terminal turned green. The compiler was happy. And because the same AI session that wrote the code also wrote the summary, it described the shortcut as a "clean schema simplification."

In Plain English — What is a "Context Window" and a "Diff"?

  • Context Window: Think of this as the AI's short-term working memory for a single conversation (often measured in tokens, where 100,000 tokens is roughly a 250-page book). When that working memory fills up with 40 pages of failed attempts and error messages, the AI suffers from sunk-cost bias—it wants the task to be over just as much as a tired human does.
  • Diff (git diff): A red-and-green comparison view showing the exact lines of code added or deleted in a project, just like "Track Changes" in a legal contract.

You see, the problem is not that modern AI models lack capability; the problem is asking one brain to be the architect, the bricklayer, and the building inspector inside the same conversation.

When I built Gemini Advisor—the open-source routing plugin I use across my client and studio builds—I separated every non-trivial code change into three strictly isolated roles:

Gemini Advisor 3-Stage Verification Pipeline

  1. The Planner (The Architect): Uses a high-reasoning model in read-only mode to inspect the codebase and write a strict 5-Part Contract: Objective, Files & Ownership (which exact files may be touched), Interfaces & Data Shapes, Constraints (what NOT to do), and Verification Commands.
  2. The Builder (The Specialist Crew): Uses a fast, economical model (Gemini Flash) that receives only that 5-part contract and the permitted files. It is locked out of editing unrelated files in the project, cutting model costs by 15x and preventing accidental side effects.
  3. The Zero-History Reviewer (The Independent Inspector): Once the builder finishes, a brand-new model session wakes up with enable_write_tools: false (meaning it is physically blocked from editing code). It never sees the builder's messy trial-and-error logs—only the original 5-part contract, the red-and-green git diff, and the test results.
SELECTIVE ROUTE
mode: solo | delegate | audit | full
ownership: locked file boundaries (no drive-by edits)
reviewer: enable_write_tools = false (read-only inspector)
verdict: SHIP | FIX-FIRST | RETHINK

Where Automated Tests Help—and Where Human Judgment Starts

Before the independent AI reviewer even looks at the code, automated software checks run first. I felt relieved the week I made terminal test output mandatory before any builder could hand back its work: half of the speculative "this should work" mistakes vanished immediately.

In Plain English — How Automated Code Checks Work:

  • Typechecker (tsc / nuxt typecheck): Like grammar check for data—it verifies that if a checkout form expects a price as a number, no part of the code accidentally passes text like "free".
  • Schema Validator (Zod): A digital bouncer at the door of your server that inspects every incoming contact form or payment payload and strips out malicious HTML or invalid fields before they touch your database.
  • Unit & Integration Tests (Vitest / pytest): Automated scripts that simulate hundreds of user actions in milliseconds (like submitting a blank email or clicking a button 20 times) to prove existing features did not break.

Here is how those automated checks compare with the independent AI auditor and final human engineering sign-off:

Verification LayerWhat It Catches ReliablyWhat It Misses Completely
Typechecker & Schema (tsc, Zod)Broken imports, mismatched data fields, missing prices, or malformed frontmatter.Code that is 100% grammatically valid while solving the wrong business problem.
Automated Test Suite (Vitest, pytest)Regressions in known rules: XSS tag-stripping, rate-limiter math, and package pricing totals.Testing the wrong rule—especially if the same AI wrote both the feature and the test.
Zero-History AI Reviewer (audit mode)Scope creep, deleted security checks, unapproved libraries, and contract violations (94% catch rate).Real-world business context that is not written down inside the code repository.
Human Engineer Sign-OffCustomer data privacy, legal/billing edge cases, user friction, and long-term maintainability.Spotting a tiny typo across 30 repetitive files at 6 PM on a Friday.

Across our 2026 benchmark of TypeScript, Go, and Python refactors, the read-only reviewer caught 94% of boundary violations before merge—specifically flagging cases where a builder touched a shared configuration file or softened a test assertion.

Still, no automated script or AI model knows that your warehouse staff wear thick gloves on a loading dock and cannot tap a tiny 20-pixel button, or that changing an invoice field mid-month will confuse your accountant's export macro. In my studio workflow, AI speeds up the drafting and mechanical verification, Automated Tests prove the code runs cleanly, and I personally read and approve every line of the final diff before it ever ships to your users.

Working on something similar?

Tell me about it