How honest are our confidence numbers?
Every AI prediction carries a confidence percentage. This page asks one question of that number: when the system says 70%, is it right about 70% of the time? The answer is measured from graded predictions — nothing estimated, nothing invented. Where measurement is thin or not yet possible, we write "n/a" instead of guessing.
Current calibration snapshot
Reliability curve — raw confidence vs reality
Each point below is a bucket of graded predictions. The horizontal position is what the raw model claimed; the vertical position is how often those predictions were actually right. A perfectly calibrated system sits on the dashed diagonal. Ours does not — and we show that.
Solid line: raw confidence buckets (claimed → measured). Dashed diagonal: perfect calibration. Dots: individual buckets, dot size roughly proportional to how many graded predictions sit in the bucket.
| CLAIMED (RAW) | MEASURED HIT RATE | GRADED N |
|---|---|---|
| loading measured data… | ||
Before and after calibration
Calibration does not make the model smarter — it makes the number it prints honest. The measured effect of remapping raw scores through the isotonic map (fit on graded history, verified on a holdout the mapping never saw):
| METRIC | BEFORE (RAW) | AFTER (CALIBRATED) | WHAT IT MEANS |
|---|---|---|---|
| Expected Calibration Error (ECE) | — | — | Raw confidence claimed far more than it delivered; the calibrated number sits close to measured reality. |
| Brier score | — | — | Overall probability quality; 0.25 is what a pure coin flip scores. The calibrated number is honest about that. |
| Band coverage (target 90%) | — | — | Share of outcomes that fell inside the published uncertainty band on the holdout. |
The measured truth, from the calibration engine: raw confidence carried almost no signal — buckets claiming 60–90% were right roughly half the time. Calibration makes the printed number honest (it now says "close to a coin flip" when that is what the data supports); the trading edge comes from the model, not from the confidence label.
Monthly history
Month-by-month measured record of calibration quality. History begins when this measurement pipeline started publishing; months without enough graded rows are written as n/a. Per-month band coverage is n/a by definition of the aggregate — conformal coverage needs row-level pairs and is only measured on the holdout snapshot above.
| MONTH | GRADED N | HIT RATE | ECE RAW | ECE CALIBRATED | BRIER RAW | BAND COVERAGE |
|---|---|---|---|---|---|---|
| loading measured data… | ||||||
Methodology — in our own words
- What is graded. A prediction is "graded" only when its outcome has been checked against the actual market move and stamped CORRECT or WRONG in the prediction log. Predictions still awaiting grading contribute nothing — they are never counted as wins or losses.
- How ECE is measured. Graded predictions are grouped by stated confidence into ten buckets. In each bucket we take the gap between the average claimed confidence and the measured hit-rate, then average those gaps weighted by how many predictions sit in each bucket. Zero means the stated confidence exactly matches reality.
- How Brier is measured. For every graded prediction we square the distance between the stated probability and the actual outcome (1 for correct, 0 for wrong) and take the mean. A permanently uniform guess scores 0.25; lower is better.
- How the calibrated number is made. A remapping is fit on past graded predictions so that raw scores land closer to their measured hit-rates, and a statistical band around that number is sized so it should contain roughly 9 out of 10 outcomes. The mapping is verified on graded predictions it was not fit on — the holdout numbers above come from that verification.
- What calibration is not. It changes no model, no signal, no trade decision. It rewrites only the confidence label. If the calibrated confidence reads ~0.49, that is the map telling the truth about limited signal — not a failure of the remapping.
- Honest gaps. Monthly history covers only months with graded data. Monthly band coverage is n/a because the aggregate cannot recompute row-level band membership; it is published only for the holdout snapshot. Every figure here is stamped with its source and date.
FAQ
What is confidence calibration?
It is the check that a stated confidence matches reality: predictions labelled 70% confident should be right about 70% of the time. Calibration remaps the stated number against graded outcomes so it stops overclaiming.
What is ECE?
Expected Calibration Error: group graded predictions by claimed confidence, measure the gap between claim and reality in each group, average the gaps. Zero is perfectly honest; our raw number was not, and this page shows the measured fix.
Why does VedKuber publish its calibration error?
Because an unverifiable confidence number is marketing, not measurement. Publishing the error monthly — including unflattering months — lets anyone check whether our numbers deserve the trust we ask for.
Why is some coverage "n/a"?
Band coverage needs each individual prediction's outcome compared against its own published band. Monthly aggregates cannot reconstruct that row-level comparison, so we publish n/a rather than an invented number. It is measured on the holdout snapshot, where it is 92.78% against a 90% target.
Disclaimers
- Backward-looking: all numbers describe past graded predictions and are not predictive of future results.
- Aggregates only: no user-level data appears on this page, ever.
- Fail-soft: if the measurement feed is unreachable or stale, we say so rather than showing stale numbers as current.
- This is a transparency report, not investment advice. Trading in securities involves risk of loss.
Related: Monthly Transparency Report · AI Transparency & Live Reports · Track Record · Rating System