Same Business Plan, Different AI Models — How Often Does It Actually Catch the Number Errors?

#Experiment design — same plan, multiple models, multiple runs

The setup is simple. We prepared one business plan in two versions: the original, with its real numbers intact, and a tampered version with deliberately inflated revenue projections. We ran both versions through four models — Haiku 4.5, Sonnet 5, Opus 4.8, and GPT-5.5 — twice per model per version (4 models × 2 versions × 2 runs each = 16 runs total).

• We left one real, pre-existing calculation error untouched in the original
• We added a deliberately inflated revenue-related error to the tampered version
• We ran each version twice per model, to also check whether the result stayed consistent run to run

What we checked was whether the calculations written in the plan actually hold up — multiplication relationships like quantity × unit price = total, or whether figures scattered across different sections are internally consistent. This doesn't mean every number in a business plan gets verified.

#Scaled-down judgment-direction test — comparing average rank order by pre-assigned grade

Code can confirm whether a calculation is correct. But judging that 'this plan is genuinely better than that one' is a different problem. Separate from calculation accuracy, we also checked scoring direction on a small, pre-graded sample.

We took 10 business plans that OpenSeed had pre-classified, by internal criteria, into strong / average / weak (3 strong, 4 average, 3 weak — assigned independently of any model's scoring), had each model score them, and checked whether the grade-level averages came out in the same rank order (strong > average > weak). Scoring ran across five lenses — market, product, team, finances, and risk — and a deterministic weighted sum produced the composite score.

We didn't use the absolute scores to compare performance across models, since they're affected by each model's own scoring tendencies and calibration differences — we only checked whether the rank order across grades held up within each model individually. This result alone can't tell us whether any individual document was correctly classified into its grade tier, or how much score overlap exists between tiers.

Across these 10 documents, all four models produced group averages in the same direction as the pre-assigned grades. This is not a validation of the judgment performance of OpenSeed's actual multi-agent system as a whole.

#So this is the combination OpenSeed uses

The reason for running this experiment is simple: to check whether the model combination OpenSeed currently uses — Sonnet 5 for the 15 specialist agents, Opus 4.8 for the Chief that synthesizes the report — still holds up as a reasonable choice against real cases.

As noted above, we didn't use absolute scores to compare models — and the strong-vs-average gap observed in these 10 documents (19.8 points for Sonnet 5, versus 6.7–14.7 points for the other models) is likewise shaped by each model's own scoring scale, so it's hard to treat as a rigorous performance comparison. Still, the 15 specialist agents exist to judge subtle differences in market fit, technical merit, and financials, and the fact that Sonnet 5 showed the largest score gap in this sample was a limited observation that happens to line up with the current setup.

The reason Opus 4.8 runs the Chief is unrelated to this experiment. OpenSeed's Chief doesn't generate a new score — its job is only to synthesize the 15 agents' judgments into a report, while the final score and verdict come from a separate, deterministic calculation. We placed Opus 4.8 there based on a separate architectural decision that prioritized long-context synthesis over scoring tendencies.

Haiku 4.5 and GPT-5.5 also reproduced the correct rank order on these 10 documents. This experiment isn't grounds for locking in this combination permanently — it's a limited check on whether an existing architectural choice still holds up against real cases. This observation fed into the Sonnet 5 placement; the Opus 4.8 placement was decided by the Chief's role design, not by any scoring result.

#Results — 3 ways OpenSeed builds trust

It's easy to say 'we check absolutely everything.' OpenSeed chose a different approach: clearly showing what was checked and what wasn't. Rather than claiming to have verified everything, we cross-check the calculations within a document, compare currently supported line items against official sources, and mark, in the report itself, exactly how far our verification actually reached.

All three run on the same principle: don't pretend to know more than you do — say only that you've verified exactly as much as you've verified.

#Why this matters if you're the one writing the business plan

Numbers are one of the first things reviewers get suspicious of in a business plan. Market size, revenue projections, growth rate — if these don't add up internally, credibility collapses right there. And even when every number checks out, whether the plan holds together as a logically persuasive whole is a separate question entirely.

• Whether the revenue projection and the market-size formula are consistent with each other
• Whether the numbers presented are used consistently throughout the document
• Whether any figure has been unintentionally inflated

Of these, the calculations written into a plan — multiplication relationships like quantity × unit price = total, or whether scattered figures line up internally — are things OpenSeed doesn't leave to AI's subjective judgment alone; a separate calculation cross-check procedure verifies them again. Whether the plan is persuasive as a whole, on the other hand, is a matter of synthesizing judgment across multiple review lenses, and the scaled-down experiment above shows only this much: applying one scoring procedure, all four models produced group averages pointed in the same direction as the pre-assigned grades.

#Why OpenSeed is publishing this

When OpenSeed evaluates a business plan, we look at the evidence behind it, not just a score. And we believe the process that produces that evidence has to be verifiable in its own right. The three trust-building procedures above are what happens when we apply that same principle to ourselves.