Green Tests Aren’t Enough: How We Review AI-Generated Code

One of the easiest mistakes to make with AI-generated code is also one of the most reassuring:

The build passes.

The types check.

The tests are green.

The feature works.

Ship it.

Those signals matter. We want all of them.

We just do not consider them sufficient.

Coding agents have become very good at producing implementations that satisfy a local task. At the same time, the industry is discovering that reviewing agent-generated software requires looking beyond whether a patch produces the expected output. GitHub has expanded its own AI code review toward broader repository context and “architectural integrity,” while OpenAI has described mechanically enforcing architectural boundaries and invariants in an agent-heavy codebase rather than relying on documentation or tests alone.

That aligns closely with a lesson we have learned while building StackLiberate:

A test can prove that an observed behavior works under the conditions it covers. It cannot, by itself, prove that the behavior was implemented in the right place.

That difference shapes how we review AI-generated code.

Correct behavior and correct architecture are different questions

Consider a feature that needs to update some project state.

An agent may find a component where the value is displayed, introduce local state, update the UI, add tests, and produce exactly the visible behavior requested.

Every assertion can pass.

But suppose the application already has an authoritative state model responsible for persistence, reload, version history, and another representation of the same project.

The new implementation works.

It is also wrong.

It has created a second place where the system believes the truth lives.

Tests written around the new component may never reveal that.

This is the class of issue we care about most during review.

Not:

Does this function return the right value?

but:

Does this implementation still respect how the system is supposed to work?

Both matter. They operate at different levels.

Agents are unusually good at locally reasonable shortcuts

This is not because coding agents are careless.

Often the opposite is true.

The dangerous implementation is frequently clean, typed, tested, and easy to explain. The shortcut emerges because the agent is optimizing from the context it currently has.

If a behavior can be implemented by:

• adding a local state variable,
• introducing a small helper,
• duplicating an existing transformation,
• special-casing one rendering path,
• or bypassing a more complicated shared abstraction,

that solution can appear entirely reasonable.

The problem becomes visible only when the reviewer understands the larger lifecycle.

What happens after reload?

What happens during restore?

What happens in another route?

Who owns this state?

Does another writer exist?

Does export reconstruct the same result?

Does this bypass an authorization boundary?

Did we solve the general contract or only today's visible case?

The resulting code may contain no obvious “bug.” The architectural error is that the new behavior belongs somewhere else.

Recent research reinforces the distinction between generating tests and establishing broader correctness. A 2026 empirical study of software-engineering agents found that simply increasing the amount of agent-written testing did not significantly improve final issue-resolution outcomes across the agents studied. Separately, a recent synthesis of empirical research found AI-generated code quality varies across correctness, security, maintainability, and complexity, with outcomes depending on specifications, human expertise, model behavior, and the surrounding workflow.

More tests can be valuable.

They are not a substitute for knowing what must be true.

We review against invariants

For substantial StackLiberate changes, review begins before implementation.

The plan usually establishes a small set of invariants: properties the feature is not allowed to violate.

An invariant might concern:

• Which representation is authoritative.
• Which layer owns a state transition.
• Whether a change must survive reload.
• Whether history and restore must reconstruct it.
• Whether stale writes must be rejected.
• Whether an existing public behavior must remain untouched.
• Whether preview and portable output must preserve the same authored result.

The implementation is then reviewed against those properties.

This gives us a stronger question than:

Did the agent implement its own plan correctly?

We can ask:

Did the resulting system preserve the contracts we agreed mattered?

OpenAI has described a similar distinction in its agent-first engineering work: rather than micromanaging every implementation, its team encodes architectural boundaries, dependency directions, structural tests, naming rules, and other invariants that agents are expected to preserve. Their argument is particularly relevant to high-throughput agent development: constraints become more valuable as implementation becomes cheaper.

That is also why we care about architecture at StackLiberate.

Not because every function needs an elaborate abstraction.

Because some boundaries should remain difficult to accidentally cross.

Tests validate behavior. Boundaries validate structure.

Traditional automated tests are excellent at questions like:

Given this input,
does this output occur?

They can also exercise integration paths, browser behavior, persistence, failure cases, and complete user journeys.

But some properties are better checked another way.

Suppose the rule is:

Only one subsystem may write this state.

You can create behavioral tests around every known route.

Or you can make the boundary explicit enough that violating it becomes mechanically detectable.

Likewise:

This runtime package must never depend on this UI layer.

That is fundamentally a structural rule.

The strongest engineering systems use several forms of evidence:

Types
   +
Static constraints
   +
Unit tests
   +
Contract/integration tests
   +
End-to-end behavior
   +
Architectural review

None replaces the others.

This is also why Kiro has pushed spec-driven and property-based approaches: the useful question is not simply whether examples pass, but whether the implementation continues satisfying properties derived from the intended behavior.

We use the same underlying idea even when the enforcement mechanism differs:

define what must remain true, then validate at the level where that property actually lives.

We look for the second source of truth

One of our highest-value review questions is simple:

Did this change create another authority for something that already had one?

AI makes duplication cheap.

If the agent cannot easily reach an existing abstraction, it can recreate enough of the behavior nearby.

Sometimes that is harmless.

Sometimes it produces two implementations that agree today and diverge six features later.

This can happen with:

• State.
• Validation.
• Sanitization.
• asset handling;
• serialization;
• routing;
• permissions;
• derived UI;
• transformation logic.

The review is therefore not just looking for duplicated lines.

We look for duplicated responsibility.

Two implementations can share zero code and still be competing sources of truth.

That is much harder for a conventional unit test to detect.

We inspect state boundaries

State-heavy products make this particularly important.

A value visible on screen may exist simultaneously as:

• Temporary UI state.
• Project state.
• Persisted state.
• A historical version.
• A rendered projection.
• An exported representation.

The existence of several representations is not necessarily a problem.

Confusing their authority is.

So when an AI-generated change touches meaningful state, we ask:

• Where does the transition originate?
• Which representation becomes authoritative?
• Which other states are derived?
• Who can write?
• What happens if two operations overlap?
• What happens after reconstruction?
• Can one projection diverge without detection?

This kind of review often catches issues that are invisible in the changed function itself.

The code looks fine because the code is fine.

The problem is the boundary around it.

We review failure semantics too

Green tests naturally encourage attention toward successful execution.

Review has to include what happens when success stops halfway.

For a meaningful operation, we want to understand:

• What remains unchanged if validation fails?
• Can an operation partially persist?
• Can a retry duplicate work?
• What happens with stale state?
• Does a late response overwrite something newer?
• Does failure leave the UI claiming something the authoritative state rejected?
• Is recovery explicit?

These questions do not mean every feature requires elaborate distributed-systems machinery.

They mean failures should have intentional semantics.

An agent can often make the happy path work very quickly.

Engineering review asks whether the unhappy path leaves a system we still understand.

AI can participate in review

None of this implies that AI should only generate code and humans must inspect every line afterward.

Agents are increasingly capable reviewers themselves.

GitHub's code-review system now retrieves surrounding repository context specifically so it can reason beyond the diff, and it allows repository-specific instructions and skills to influence review. OpenAI describes agent-to-agent review loops in which agents inspect and iterate on each other's work.

We use AI heavily in the review loop too.

An agent can be excellent at:

• Tracing affected paths.
• Finding duplicated logic.
• Checking whether a plan was followed.
• Looking for missing failure cases.
• Comparing implementation against documented constraints.
• Running validation.
• Challenging another agent's assumptions.
• Searching for unexpected blast radius.

The important distinction is that the reviewer needs an independent question to answer.

Asking:

“Review your work and tell me whether it is good.”

is weaker than asking:

“Verify that this implementation preserves these five invariants and identify every path that can mutate this state.”

Review quality improves when the review has a contract.

The same is true for humans.

Human review moves up a level

As models improve, we expect less value to come from humans manually checking every syntactically obvious line.

Martin Fowler recently described a useful distinction between vibe coding and agentic programming: in the latter, humans remain responsible for what the software does and how it works even when agents produce the code.

That describes our direction well.

Human attention is scarce.

We would rather spend it asking:

• Is this abstraction necessary?
• Is the responsibility in the right place?
• Did the implementation widen scope unnecessarily?
• Is a shortcut being mistaken for a contract?
• Are we preserving the system's existing invariants?
• Are the failure semantics acceptable?
• Will someone—or another agent—be able to reason about this six months from now?

Those questions have more leverage than debating whether a helper should have been written with three lines instead of five.

AI can generate a great deal of code.

The human role increasingly involves deciding which structures deserve to survive.

A green build is a checkpoint, not a verdict

We like green builds.

They tell us something important.

The implementation compiles. Known behaviors hold. Automated checks did not detect a regression within the properties they cover.

That is evidence.

Then review continues.

For meaningful changes, our mental model looks closer to:

Does it work?
      ↓
Do the tests prove the intended behavior?
      ↓
Does it preserve the agreed invariants?
      ↓
Is state owned in the right place?
      ↓
Are failure boundaries still explicit?
      ↓
Did we add a shortcut that becomes tomorrow's architecture?
      ↓
Can the system still be understood?

Only the first two questions are primarily about green tests.

The rest are about engineering.

AI makes this more important

The cost of producing plausible code has fallen dramatically.

That is a remarkable advantage.

It also means we can create architectural debt faster than before.

A weak abstraction no longer requires an engineer to spend two days implementing it. An agent can build it, document it, test it, and integrate it before lunch.

The natural response is not to slow the agent down until it behaves like an old development process.

It is to move more judgment into the places where judgment has the highest leverage:

before implementation, at architectural boundaries, and during acceptance.

For StackLiberate, green tests are part of the bar.

They are not the bar itself.

We want the feature to work.

We also want it to belong.

And increasingly, that distinction is where the most important engineering happens.