Intro.
#What Calibration Is — An Analogy
Say a weather service forecasts an 80% chance of rain tomorrow. If it actually rained on 80 of the 100 days it made that same 80% forecast, the forecast is well calibrated. If it only rained on 50 of those 100 days, the forecast is overconfident.
TIP
Calibration measures whether an AI is actually right as often as it sounds confident. It's not about whether the score itself is accurate — it's about whether the probability implied by that score is accurate.
02
#Applying This to Business Plan Evaluation
When an AI tells a business plan it has an '85% chance of success,' the AI is well calibrated only if roughly 85 out of 100 plans that received that same score actually succeed. The bigger the gap, the stronger the signal that the AI's confidence is miscalibrated.
| AI-predicted success probability | Actual success rate — well calibrated | Actual success rate — overconfident |
|---|
| 50-60% | ~55% | 20% |
| 60-70% | ~65% | 30% |
| 70-80% | ~75% | 45% |
| 80-90% | ~85% | 60% |
03
#ECE — The Metric for Measuring Calibration
Expected Calibration Error (ECE) is a metric that averages the gap between an AI's predicted probability and the actual outcome. The closer it is to zero, the better calibrated the AI is, and it's used as a standard measurement in academic research. This is a well-established, publicly documented concept in the machine learning field.
- ECE < 0.1 — well calibrated (predictions match reality within an average of 10 percentage points)
- 0.1 ≤ ECE < 0.2 — needs recalibration
- ECE ≥ 0.2 — over- or under-confident; interpret results with caution
주의
Because ECE is an average, a large error concentrated in just one score range can still produce a small overall number. You need to look at a per-bucket calibration chart alongside it for an accurate diagnosis.
04
#Limitations Users Should Know About
Calibration is a well-established academic concept, but it comes with two limitations. First, measuring it requires enough validation data. Second, as markets and industries shift, past calibration needs to be re-verified to confirm it still holds today.
- Validation data — needs at least dozens to hundreds of matched 'prediction vs. actual outcome' pairs
- Time dependency — calibration measured against last year's market doesn't necessarily hold this year
- Industry variation — calibration is best measured separately for each industry, whether IT, biotech, or D2C
Summary.
#Where OpenSeed Stands
OpenSeed follows the principle that AI review results must be validated against human evaluators and real-world outcomes. We accumulate business plan validation data — fundraising and grant-program results — to measure how well our AI review's results line up with reality, and we disclose that measurement transparently on our admin dashboard.
CTA
OpenSeed's AI review is under the current payment terms shown on the analysis start screen. We provide the meaning behind every score, together with the reasoning that produced it.
The Meaning Behind Every AI Score, With Reasoning Included
Get your score and the reasoning behind it, under the current payment terms shown on the analysis start screen.
Pre-submission business-plan review with clear service limitations
Start OpenSeed Review →