How close is an assistant’s memory to a recorded sale?
Ask a chat assistant what a domain is worth and it answers from whatever it recalls, with no citation and no interval. This page measures that recall against 218 recorded sales - the same published blind holdout this site scores its own deterministic engine against at /predictions/, using the identical metrics.
The fixed point every row here is measured against
audit.domains deterministic engine (this site, same holdout): 31.65% within 2x of the recorded sale, median error 84.6%, over 218 domains. Computed with each domain's own sale withheld, identically to /predictions/. Not an assistant answer - the fixed point every row above is measured against. These are the same currently-published figures as /predictions/, printed here unchanged whether or not a submitted model beats them.
Witnessed
Run by the maintainer or a steering-group member, who holds the raw transcript log.
No witnessed submissions yet.
Self-reported
Bring-your-own-transcript submissions. A signed log proves only the bytes the submitter chose to upload; these rows are never merged into the witnessed table above.
| Model | n | Within 2x | Median APE | Bias |
|---|---|---|---|---|
| Claude (Anthropic), memory only | 218 | 10.55% (7.13-15.33) | 87.3% | 7.69x under, median |
Name by name
The research report Memory vs the market sets each answer beside the recorded sale and this engine’s estimate, with charts, segments, the largest misses and the per-name data.
Play against these answers
Beat the Hunch deals five of these 218 names a day. You price each one, then see the recorded sale next to the self-reported assistant answer above and this engine’s withheld estimate, scored by how many times off each figure was.
Submit a result
The question set is fixed and versioned (odvs-a-prompts-v2, sha256 501c44c976d5…): 218 prompts, each asking for a single point-estimate dollar figure for one domain, with the recorded sale price withheld. Run them through any assistant on your own account and key - this project never calls a model API itself, so there is no server-side cost and no vendor added by running this benchmark. Full steps, the submission format, and the scoring script are in tools/odvs-a/README.md in the source repository. A submission is scored with the same code this site gates its own accuracy on, and a result that beats this engine is published exactly like one that does not.
Method
sales reported by DNJournal and Sedo, kept out of the corpus the engine draws its evidence from. audit.domains out-of-sample predictions, Recent sales slice (n=218), methodology revision 4.12, evaluated 2026-09-23. Each estimate was produced with that domain's own sale withheld. Quote a row with its basis, and the table with its n. Metrics are computed with the same code as the engine’s own accuracy gate (median absolute percentage error, share within 2x with a Wilson 95% interval, and the median ratio as a signed bias figure); no number here is a probability, and no single accuracy percentage is quoted without its sample size. The same rows are machine-readable at /accuracy/assistants.json (schema: tools/odvs-a/results.schema.json).