There is a seductive idea in the AI development community — one that feels so intuitive it rarely gets questioned: take a problem, send it to the best AI model available, rate the output honestly, feed the rating back, and iterate. Do this enough times and the quality converges toward perfection. Keep looping. Eventually you reach 100/100.
Then a second idea follows naturally: if one model plateaus, send the same problem to multiple frontier models. Compare their outputs. Merge the best parts. Surely the combined intelligence of the world's most powerful language models can produce something production-ready.
We tested both ideas with a real engineering problem, honest evaluation criteria, and no thumb on the scale. The AI improved fast. Then it stopped. And the gap between where it stopped and where production-ready begins turned out to be a gap of a fundamentally different kind of knowledge.
But here is the part that makes this genuinely interesting: senior engineers don't reach 100/100 either. Neither does Google. Neither does Stripe. The question "can AI reach perfection?" turns out to be the wrong question — because perfection, in software, may not exist. And understanding why changes how you think about AI trust entirely.
The Problem: Payment Processing
We chose payment processing deliberately — not because it is the hardest engineering problem, but because the consequences of being wrong are measured in money. Not in aesthetics or style debates. In actual financial discrepancies that show up in ledgers, customer complaints, and compliance audits.
The prompt was identical across all models: design and implement a payment processing module for an e-commerce platform. Handle card payments through an external gateway. Support refunds. Enforce idempotency. Manage payment state transitions. Handle partial failures. Produce an audit trail suitable for financial compliance. The rubric was designed by a senior payment engineer before the experiment, covering correctness, error handling, security, idempotency, state management, observability, edge case coverage, and production readiness. What follows is an internal case study, not a controlled scientific experiment. The scores are our honest assessments against that rubric. What matters is not the specific numbers but the patterns they reveal.
GPT-5: The Loop
We started with GPT-5.
Round 1 — Score: 74/100. Impressive for a first draft. Clean structure, input validation, gateway integration, basic error handling. What it missed: no idempotency key. Currency stored as floating-point. No timeout on the gateway call. No state machine for payment transitions. Error messages that leak internal details. No consideration of partial failures — authorization succeeds, capture fails, and the system has no defined recovery path.
Round 2 — Score: 83/100. Added idempotency keys, switched to integer cents, added a gateway timeout, introduced retry logic with exponential backoff. A simple audit log appeared.
Round 3 — Score: 88/100. A state machine for payment transitions. Proper audit trail. The authorization-but-capture-failed case now had a recovery path. Connection pooling. Better error types.
Round 4 — Score: 91/100. Circuit breaker for the gateway. Request tracing. Cleanup mechanism for abandoned payments.
Round 5 — Score: 92/100. Minor refinements. Improved documentation.
Rounds 6–8 — Score: 92/100. 92/100. 92/100. Code reorganization, better naming, rewritten comments. The score did not move. GPT-5 had exhausted its ability to improve this code.
Claude Opus: The Same Problem, Different Ceiling
Round 1 — Score: 71/100. Less complete than GPT-5's first output — but more honest. It explicitly commented on areas where the implementation was simplified and would need production hardening. Less code, more self-awareness about its own limitations.
Round 2 — Score: 85/100. Claude jumped further on the second iteration. Rather than patching individual issues, it redesigned the error handling architecture. The state machine appeared in round 2, not round 3.
Round 3 — Score: 89/100. Better reasoning about edge cases. The retry logic included a specific note about the danger of retrying after a timeout when the original request may have succeeded — a subtlety GPT-5 never surfaced.
Round 4 — Score: 91/100. Strong state machine with explicit transition guards. Domain-specific failure categories.
Round 5 — Score: 93/100. The most interesting round in the entire experiment. Claude caught something no other model caught: the case where the payment gateway returns HTTP 200 but the response body contains a decline. The HTTP layer succeeds; the business layer fails. Claude added explicit response-body validation. This is a real production failure mode that causes silent payment failures.
Rounds 6–8 — Score: 93/100. 93/100. 93/100. Plateau.
Gemini 2.5 Pro: The Third Perspective
Round 1 — Score: 76/100. The highest first-round score — the most complete initial output. But shallower: more features, less depth in each.
Rounds 2–4 — Scores: 84, 88, 90. Solid improvements, approaching GPT-5's quality. The plateau began earlier.
Rounds 5–6 — Score: 91. 91. Ceiling reached at 91.
The Convergence Graph
Score
100 ┤ ← 100/100
│
96 ┤ ← senior engineer + peer review
│
93 ┤ ─── ─── ─── ─── Claude Opus ceiling
92 ┤ ─·─ ─·─ ─·─ ─·─ ─·─ GPT-5 ceiling
91 ┤ ··· ··· ··· ··· ··· ··· Gemini ceiling
│ ·
88 ┤ · ·
│ · ·
84 ┤ ·
│ ·
76 ┤ · ← the gap:
74 ┤ · every production-critical
71 ┤· concern lives here
│
├──┬──┬──┬──┬──┬──┬──┬──┬──┬──┬──
1 2 3 4 5 6 7 8 9 10 11
Iteration
─── Claude ─·─ GPT-5 ··· GeminiThree models. Three different ceilings. One identical pattern: fast initial improvement, then a long plateau well below 100. The problems remaining in the 93-to-100 gap are qualitatively different from the problems solved between 71 and 93.
What Each Model Caught That Others Missed
Issue GPT-5 Claude Gemini
────────────────────────────────────────────────────────────
HTTP 200 with body decline ✗ ✓ ✗
Retry-after-timeout danger ✗ ✓ ✗
Self-documented limitations ✗ ✓ ✗
Most complete first draft ✗ ✗ ✓
Structured error categories ✓ ✓ ✗
Circuit breaker pattern ✓ ✗ ✓
Cleanup for abandoned payments ✓ ✗ ✗
Request tracing ✓ ✗ ✓
────────────────────────────────────────────────────────────
Race condition on balance ✗ ✗ ✗
PCI DSS considerations ✗ ✗ ✗
Void vs. refund distinction ✗ ✗ ✗
Duplicate charge on retry ✗ ✗ ✗
Persisted circuit breaker state ✗ ✗ ✗The top half is interesting — each model has genuine strengths the others lack. If you used only one, you would believe its architectural choices were obviously correct. The divergence is only visible when you compare.
The bottom half is devastating. Five critical production issues. Zero models caught any of them across all iterations. The bottom half is the 93-to-100 gap — where the money is lost and the compliance audits fail.
What Would a Senior Engineer Achieve?
We showed the problem to the senior payment engineer who designed the rubric and asked her to build her own implementation from scratch, without AI assistance.
Her first pass: 94/100. She started where the AI plateaued. Not because she writes faster — she took two hours where the AI took seconds. Because she started with the architecture that matters: the state machine, the idempotency boundary, the failure recovery strategy. She wrote those first. In our experiment, the AI consistently wrote the happy path first and progressively added failure handling through feedback — structuring the solution around what goes right, then patching what goes wrong. She structured it around what goes wrong from the beginning.
She caught the race condition immediately — it was the second thing she addressed, before writing the main processing logic. She used integer cents without being told. She included a PCI compliance comment block. She distinguished between void and refund. She handled the retry-after-timeout scenario. These were not conscious decisions. They were things she could not have written the code without — because she has lived through the consequences of each one.
After her own review: 96/100. She found two issues — a missing edge case in timeout handling and an audit log entry missing the idempotency key.
After peer review: 97/100. A colleague found a concurrency issue in the cleanup mechanism.
After integration testing: 98/100. Testing revealed a failure mode in the interaction between the circuit breaker and the retry logic that neither engineer had anticipated.
After six months in production: still finding things. Two edge cases surfaced — one related to a specific gateway's non-standard timeout behavior, one related to daylight saving time affecting settlement windows.
The senior engineer started at 94. The best AI plateaued at 93. The distance is one point — but the nature of that point matters enormously. Her 94 included the race condition, the PCI awareness, the void-vs-refund distinction. The AI's 93 did not. The gap is not quantitative. It is structural — the difference between code that handles everything the developer anticipated and code that handles everything the developer has experienced going wrong.
Does 100/100 Exist?
Everyone in this experiment — every AI model, the senior engineer, the peer reviewer, the testing process, months of production — approached 100/100 without reaching it. That raises a deeper question: is 100 a destination, or a direction?
100/100 for a payment system would require: every possible input handled correctly (infinite). Every failure mode covered (not enumerable). Every regulation satisfied (they change). Every concurrency scenario handled (combinatorially explosive). Security against every attack (the attack surface evolves). Performance under every load condition (includes adversarial conditions).
If you define 100/100 as "the code does exactly what the spec says" — that is achievable, with enough effort. If you define it as "the system behaves correctly under every condition it will ever encounter" — that is a direction, not a destination. It is the asymptote that both AI and human engineers approach but structurally cannot reach, because the definition of "correct" is not static. Requirements change. Threats change. Dependencies change. The world changes around the software while it runs.
Even formally verified software — seL4, CompCert — is verified against a specification. The proof shows the implementation matches the spec. It does not prove the spec captures all of reality. If the specification is incomplete — and every specification of a non-trivial system is incomplete — the verification proves a correct implementation of an incomplete description.
This matters for the AI trust question because people ask "can AI reach 100?" as though 100 is a place humans have already been. Humans have not been there either. The question is not "can AI reach perfection?" It is "how far can AI get, at what speed, and what is the nature of the remaining gap?"
"Good Enough" Depends on What Breaks
The experiment treats 100/100 as a single target. In practice, the acceptable quality threshold is not one number — it is a function of the consequences of failure.
System Cost of failure Reasonable threshold
──────────────────────────────────────────────────────────────────────
Personal landing page Very low Basic testing
Internal dashboard Low–medium Automated tests + review
E-commerce storefront Medium Testing + monitoring
Payment infrastructure High Deep review + security + compliance
Healthcare system Very high Extensive validation + certification
Aircraft control software Extreme Formal verification + regulatory certAI-generated code that scores 92/100 may be entirely sufficient for an internal dashboard. That same 92/100 is negligent for a payment system. The score didn't change — the consequences did.
This means the correct response to "can we trust AI?" is not a universal yes or no. It is: what is the cost of being wrong for this specific system, and does the current level of verification match that cost?
AI doesn't change this principle. It changes where on the development timeline the verification bottleneck sits. When code takes weeks to write, verification is a fraction of the effort. When code takes minutes to generate, verification becomes the dominant cost — and the dominant skill. The question shifts from "how quickly can we build it?" to "how quickly can we establish that it deserves to exist?"
That shift has profound implications for how engineering teams are structured, what skills are valued, and what it means to be a software professional.
Trustworthiness Is Temporal
There is a dimension of quality that the experiment cannot capture: time.
A system that was reviewed perfectly on day one becomes less trustworthy over time — not because the original code degraded, but because the world changed around it. Two years after deployment: dependencies have updated with breaking changes. Browser APIs have shifted. The payment gateway has modified its response format. Regulations in three jurisdictions have been revised. Attack techniques that didn't exist at launch are now automated. Traffic patterns have changed. New integrations were added by different developers. The operating system patched a behavior your code relied on.
Trustworthiness is not a binary — trusted or not trusted. It is conditional: trusted under a particular set of assumptions, at a particular point in time, for a particular set of conditions. Those assumptions erode continuously. A system that was 98/100 at launch may be 94/100 eighteen months later without a single line of code changing.
This means quality assurance is not a phase. It is a continuous process. Launch is not the end of QA — launch begins another phase of QA: operational evidence. Real users introduce unexpected sequences, unusual data, concurrency patterns, geographic differences, and edge cases that never appeared in staging. Production is where the software meets conditions that no prompt, no rubric, and no review process anticipated.
For AI-generated code, this has a specific implication: the code may pass every check at generation time and still accumulate risk silently as the environment changes. Continuous re-validation — automated monitoring, anomaly detection, periodic re-review — is not optional. It is part of the cost of operating software, regardless of who or what wrote it.
The Loop and the Consensus: Diminishing Returns
In this experiment, the pattern was clear. Iterations 1 through 3 produced the large majority of meaningful improvement. Iteration 4 provided the last structural addition. Everything after that was polishing.
For practical purposes: three to four iterations is the useful range. After that, you are paying the cost of evaluation for essentially zero quality gain. The model has applied every relevant pattern from its training data. Additional iterations produce reorganization, not improvement.
We also tried combining models. A merged best-of-all solution — giving all three outputs to a fourth model for synthesis — scored 90/100. Lower than any individual plateau. The merger chose the most common design decisions rather than the best. It eliminated Claude's response-body validation because it appeared in only one output. And it created a new integration bug by combining incompatible subsystems.
A sequential chain — each model improving the previous model's output — converged toward average. By model 3, the distinctive strengths of earlier models had been smoothed away. The result was polished, readable, and less robust than the best individual output.
The most valuable insight in the entire experiment — Claude's HTTP-200-with-decline detection — was eliminated by both consensus approaches. The thing that makes an insight valuable — that almost nobody thinks of it — is the same thing that makes consensus discard it.
"It Was the AI" — The Accountability Problem
Suppose you ship the AI-generated payment module. It scores 92/100. It passes code review. Three weeks later, a customer is charged twice due to the retry-after-timeout scenario.
Who is responsible?
"It was the AI" is becoming the software industry's newest accountability escape hatch — the 2026 version of "it was a third-party library." Using AI does not transfer the organization's responsibility for software it chooses to deploy. The code was reviewed, approved, and shipped by human beings making human decisions. The tool used to generate a first draft does not change who is accountable for what was shipped.
The aviation analogy is precise. When an autopilot errs, the crew is responsible — because the crew is expected to understand the system well enough to recognize when the autopilot is wrong. The same expectation should apply to AI-generated code: the AI is a tool; the engineer is the accountable professional.
The uncomfortable implication: the engineer must understand the code well enough to recognize the failure modes the AI missed. For a 92/100 payment module, that means understanding race conditions, PCI compliance, void-vs-refund semantics, retry-after-timeout behavior. These are exactly the skills that the "AI replaces developers" narrative calls unnecessary. They are becoming more necessary — because without them, nobody is actually accountable for what gets shipped.
The scariest scenario is not a senior engineer blaming AI. It is a junior engineer who uses AI to generate a payment module, ships it, and genuinely does not know enough to recognize what's wrong — and the organization has no review process that catches it. That is not an AI failure. It is a structural failure in engineering culture.
From Writing to Verification: The Bottleneck Shift
AI changes the economics of software creation dramatically. If generating code becomes 10x faster, the bottleneck moves. It moves from writing to verifying.
You can generate thousands of lines in minutes. But you cannot verify thousands of lines at the same speed — not with tests, not with reviews, not with compliance checks. The production pipeline doesn't get 10x faster just because the first draft appeared 10x sooner.
In our experiment, one senior reviewer examining the AI's best output was worth more than eight additional AI iterations. Not because the reviewer is smarter — because the reviewer carries a different kind of knowledge. But also: additional human reviewers beyond the first show rapidly diminishing returns, because experienced engineers share similar mental models of what to look for. A second senior reviewer will overlap substantially with the first, while still bringing a different set of experiences and blind spots.
The economically rational approach is: three to four AI iterations to exhaust pattern-based improvement, then one senior human review to catch experiential issues, then testing, then production monitoring. That combination reaches further than either approach alone for a fraction of the total cost.
And there is an important forward-looking question: what happens when AI becomes better at verification than at generation? Imagine a system where AI generates code, then a separate AI adversarially attacks it, a third generates tests, a fourth monitors production behavior, and a fifth detects anomalies — while humans define acceptable risk thresholds and govern the process. The architecture becomes: AI generates, AI challenges, AI monitors. Humans govern risk. That future is not here yet — but it is the direction worth building toward, and it is considerably more interesting than the current question of whether AI can replace individual developers.
The Psychology of the Loop
Why does iterating with AI feel so powerfully convincing? Three cognitive mechanisms compound:
Familiarity as proxy for quality. By round 6, the code is deeply familiar. You have watched it evolve through five versions. Familiar things feel correct — a well-documented cognitive pattern. But familiarity with the evolution of code is not understanding of its behavior under conditions you have not tested. It is possible to become intimately familiar with a system and remain blind to its most consequential failure modes.
Effort justification. Six rounds of evaluation and careful feedback represent significant investment. If the output still needs fundamental work after six rounds, the effort feels wasted. The sunk cost creates unconscious pressure to inflate your quality assessment. This is not a character flaw. It is how cognition functions under investment.
The confidence gradient. Each iteration produces a higher score. The trajectory is upward. When scores move from 74 to 83 to 88 to 91, 95 feels like the next natural step. But the curve is asymptotic, not linear. Each additional point requires exponentially more insight — insight the model may not possess. Extrapolating from the steep part of the curve to predict the flat part is how AI-generated code gets shipped with systematic blind spots nobody noticed because the improvement trajectory was so encouraging.
The senior engineer we brought in had none of these biases. She saw the code for the first time, with no investment in its evolution, no familiarity with its journey, no trajectory to extrapolate. That is why she found problems that eight rounds of careful iteration had not surfaced. Fresh eyes are not a platitude. In the context of AI feedback loops, they may be the most important quality tool available.
The Correlated Blindness Problem
There is a hidden statistical assumption in the multi-model approach: that AI models have independent error distributions. If their errors were independent, adding more models would multiplicatively reduce undetected issues.
The errors are not independent. They are correlated — often strongly.
Independent errors (ideal):
P(all miss) = P(A misses) × P(B misses) × P(C misses)
= 0.08 × 0.08 × 0.08 = 0.000512
Correlated errors (reality):
P(all miss) ≈ P(issue underrepresented in training data)
= barely reduced by adding more models trained on similar dataThe race condition was missed by all three models — not because each independently failed, but because race conditions in payment systems are poorly represented in the code that all three learned from. PCI compliance was missed universally because regulatory constraints exist in compliance documents and auditor conversations, not in GitHub repositories.
Multi-model approaches help with errors in the well-represented part of the training distribution. They provide diminishing returns on the underrepresented part — which contains the most consequential failures. The more models you add, the more you reduce errors you would have caught anyway, and the less you reduce the ones that actually matter.
Where This Leaves Us
The feedback loop is useful — genuinely useful. Three to four iterations produce real improvement, compressing valuable engineering work into minutes. But the loop does not converge to correct. It converges to plausible. And plausible is not the same as production-ready.
Software does not converge to perfect. It converges to an acceptable level of risk — and that level must be continuously re-established as the system and its environment change. This is not an AI-specific observation. It is a property of all non-trivial software. But AI makes the pattern more visible, because AI compresses the easy part of the journey so dramatically that the hard part becomes impossible to ignore.
The intellectual architecture of this problem has layers. AI converges toward plausible. Engineering converges toward sufficiently verified. Production reveals whether the assumptions were sufficient. Time changes the assumptions. Therefore, trust is not something you establish once — it is something you continuously re-earn.
In our experiment, one senior human review was worth more than eight additional AI iterations — not because humans are smarter, but because they carry knowledge built from consequences. Use AI aggressively where its patterns are reliable. Increase verification where the consequences are high, the environment is changing, or the failure modes are poorly represented in the model's learned patterns. And do not confuse a high score on a feedback loop with readiness for conditions the loop never anticipated.
The loop is a tool. The judgment to know when the loop has given you everything it can — and what remains beyond it — is something you bring to it. That judgment is built through experience, through consequences, and through the willingness to ask not "does this look right?" but "what am I not seeing?"