There is a problem with the way we talk about AI assisted software development.
We often discuss it as though there is one activity called "coding with AI." In practice, there are several very different ways to work with a model.
You can give it a loose request and see what happens. You can explain the intent. You can give it repository context. You can write an explicit specification. You can let an agent run tests, inspect failures, and repair its own work.
Those approaches are often debated as philosophies.
I wanted to test them as engineering treatments.
So I built ADP-001, Comparing AI Development Modes, a reproducible lab that asks a simple question:
When the model stays the same, what changes when we change the way the engineering problem is represented and the way the agent is allowed to respond to feedback?
The answer turned out to be more interesting than "spec is better than vibe" or "agents are better than prompts."
The experiments suggest that reliable AI development has at least three separate dimensions:
- Information: what the model knows before it acts
- Observability: what the agent can learn from reality after it acts
- Repair budget: how much bounded opportunity it has to respond to what it learns
And the value of each depends on the problem.
The First Result Was That Nothing Differed
The first experiment used a small API rate limiting task.
I compared five treatments:
- vibe coding
- intent driven development
- spec driven development
- context driven development
- agentic development
All five passed.
Every treatment scored 1.0 with one model interaction and zero repairs.
At first that looks like a failed experiment. It was actually useful.
The task was too easy to expose a difference. A capable model could infer enough of the engineering requirements even from the loose request.
That matters because it challenges one common assumption in both directions.
It is wrong to assume that every task needs a detailed specification, a large context package, and an autonomous agent loop. But it is also wrong to conclude from a successful small task that those things never matter.
Simple problems can hide the differences between development methods.
Then the Problem Became Constraint Dense
The next experiment kept the same model but moved to a bounded TTL response cache with more subtle requirements.
The evaluator checked client isolation, write invalidation, expiry, LRU capacity, mutation safety, concurrency, response compatibility, exact TTL boundary behavior, and dependency discipline.
This time the treatments separated.
| Treatment | Full passes | Pass rate | Mean score |
|---|---|---|---|
| Vibe | 0/10 | 0% | 0.6667 |
| Intent | 0/10 | 0% | 0.6667 |
| Context | 6/10 | 60% | 0.8667 |
| Spec | 10/10 | 100% | 1.0000 |
| Agentic | 10/10 | 100% | 1.0000 |
This was the first strong signal.
But the aggregate scores still did not explain what was happening.
Vibe Coding Did Not Fail in the Way I Expected
I replayed all fifty generated candidates against the individual properties.
The vibe implementations passed almost everything.
They handled normal cache reads. They invalidated on writes. They expired entries. They isolated clients. They bounded capacity. They handled concurrency. They even got the exact TTL boundary right in all ten runs.
But mutation safety was 0/10.
Every vibe implementation allowed a caller to mutate state in a way that could corrupt a future cached response.
Intent driven development showed a similar gap. Mutation safety was also 0/10, and exact TTL boundary behavior passed only 2/10.
Context changed the picture. Mutation safety rose to 10/10, while exact TTL boundary behavior reached 6/10.
Spec and agentic passed every tested property in all ten runs.
This led me to a more precise conclusion:
AI generated code can look functionally complete while repeatedly missing subtle correctness properties that were never represented clearly enough in the engineering information.
That is a very different claim from saying vibe coding produces bad code.
In this experiment, vibe coding produced code that looked good on the obvious feature behavior. The weakness appeared in a less visible correctness property.
Repository Context Is Not the Same as a Feature Specification
The context treatment is worth looking at separately.
It received durable repository guidance: architecture expectations, engineering rules, and agent instructions.
That improved reliability from 0/10 full passes in vibe and intent to 6/10.
But it still did not reach the 10/10 consistency of the explicit specification.
This is why I think AI ready repositories need both kinds of information.
Repository context tells the model how this codebase expects software to be engineered. A feature specification tells the model what this particular change must do.
One should not be expected to replace the other.
Was Agentic Development Actually Better?
At this point spec and agentic were both 10/10.
It would have been easy to say the more advanced approaches won.
But there was a problem: the agentic treatment used zero repairs.
It was receiving strong information and getting the implementation correct immediately. That meant the experiment had not shown any value from autonomy itself.
So I built another phase where the one shot spec condition and the agentic condition received exactly the same initial information.
Both passed on the first attempt.
The result was simple:
When the information was already sufficient, autonomy added no measurable value on that task.
That may be one of the most practical findings in the entire experiment.
Before adding more agent loops, more tools, and more autonomous retries, improve what the model knows.
Static Information Is Not Always Enough
Of course, some engineering knowledge cannot be known completely from a specification.
Integration work often depends on the actual behavior of an adapter, runtime, environment, API, build tool, or deployment target.
So the next experiment created a runtime integration mismatch.
Both treatments received identical initial information. The difference was that one treatment got a single attempt while the other could inspect visible runtime failures and repair.
The one shot result scored 0.3333.
The runtime feedback loop improved to 0.6667 after three repairs.
At property level, the trajectory was even more interesting. It stayed stuck at 1/7 properties for three attempts, then jumped to 6/7 on the fourth.
A later run with eight allowed repairs never improved at all. It stayed at 1/7 through every attempt.
This is where a second principle emerged:
Autonomous iteration is not equivalent to autonomous correctness.
Giving an agent permission to keep trying is not the same as giving it enough signal to understand what is wrong.
How Much Repair Budget Is Enough?
The obvious next question was whether the agent simply needed more opportunities.
But comparing separate runs would be unfair because every run begins with a different stochastic implementation.
So I froze one starting candidate and branched several repair budgets from exactly the same implementation.
Then I replicated that design across ten independent frozen starts.
The resulting recovery curve was:
| Repair budget | Full recovery | Mean final score | Mean repair tokens |
|---|---|---|---|
| 0 | 0% | 0.3333 | 0 |
| 1 | 0% | 0.3666 | 1,733 |
| 3 | 30% | 0.6000 | 4,851 |
| 5 | 60% | 0.7333 | 6,969 |
| 8 | 60% | 0.8000 | 9,764.4 |
The important part is not that five repairs is some magic number. It is not.
The important part is the shape of the curve.
Recovery improved as the budget increased from one to three to five.
But increasing the budget from five to eight did not increase full recovery beyond 60% in this ten start sample. Mean repair token use, however, increased by roughly 40%.
The larger budget did improve the average final score, so some failed branches moved closer to correctness. But more retries did not create more complete recoveries.
More autonomy increased opportunity for recovery, but eventually the reliability benefit flattened while the cost kept rising.
Three Dimensions of Reliable AI Development
After these experiments, I no longer think the most useful question is whether a team should use vibe coding, spec driven development, or agents.
The better question is whether the workflow has enough of three things for the risk and complexity of the task.
1. Information
What does the model know before it acts?
This includes intent, constraints, architecture context, repository rules, behavioral specifications, and acceptance criteria.
If the behavior is knowable in advance, representing it clearly can be cheaper than forcing an agent to rediscover it through failure.
2. Observability
What can the agent learn after it acts?
This includes tests, compiler errors, runtime failures, policy checks, integration behavior, and deployment signals.
An agent cannot repair what it cannot observe well enough to reason about.
3. Repair budget
How much bounded opportunity does the agent have to respond?
This includes retry limits, token budgets, latency limits, termination rules, and escalation conditions.
Too little budget can stop a recoverable process. Too much budget can consume resources without increasing full correctness.
An AI Ready Repository Is More Than an Instructions File
This work also changes how I think about AI ready repositories.
It is tempting to define readiness as having an instruction file that tells Copilot or an agent how the repository works.
That is useful, but incomplete.
A mature AI ready repository should make several kinds of knowledge explicit:
- the architecture and important boundaries
- engineering rules and allowed dependencies
- the exact behavior expected from the current change
- tests and validation that expose meaningful failures
- clear stopping and escalation rules for autonomous repair
The repository is not just context for generation.
It is part of the feedback system that determines whether an agent can reason, verify, and recover safely.
What This Does Not Prove
I want to be careful with the conclusions.
These experiments used one primary model, GPT-5.6 Luna, on bounded Python tasks. Ten runs is enough to expose a useful signal, but it is not enough to turn the observed percentages into universal laws.
The labels used in the experiment are also implementations of those development styles, not official definitions for the entire industry.
And a full evaluator pass means the implementation passed the properties encoded by the lab. It does not prove the absence of every possible defect.
So I would not claim:
- vibe coding is always bad
- specifications always win
- five repairs is an optimal agent budget
- agentic development is inherently superior
What I am comfortable saying is narrower and, I think, more useful.
The Practical Decision
For a small, reversible task, a lightweight request may be enough.
For work with subtle behavioral boundaries, encode the specification.
For work inside a real codebase, give the model durable repository context as well as the feature requirements.
For integration work where reality contains information that the prompt cannot, give the agent observable validation feedback.
And when you allow autonomous repair, give it a bounded budget and an escalation path. Do not assume that unlimited retries eventually become correctness.
What Actually Improves AI Generated Software?
The experiments started with a comparison of development styles.
They ended somewhere more useful.
Reliable AI development is not produced by maximum prompting or maximum autonomy.
It comes from deliberately engineering the information available before execution, the observability available after execution, and the amount of bounded repair opportunity in between.
That gives me a simple model:
Information → Implementation → Observability → Repair → Reliability
Give the model the right information. Give the agent a way to observe reality. Give it enough opportunity to repair. Then stop when additional autonomy no longer buys enough reliability to justify its cost.
That, more than any single label, is what I think AI ready engineering needs to become.
You can inspect the complete experimental design, code, prompts, evaluators, and research report in the ADP-001 lab.
Explore this topic
Part of the AI Native Engineering topic collection.