Development notes
Measuring AI Accuracy
Prompt counts show how often a person intervened. They do not, by themselves, show whether the AI was right or wrong.
The revised conclusion
There is no honest single-number “AI accuracy” score in this project yet.
The project has strong operational evidence: prompt history, source runs, ingest reports, record failures, duplicate checks, approvals, and verification notes. It does not have a labeled gold-standard dataset that can turn all of that into one accuracy percentage.
What changed
The first measurement mixed unlike things
The original version of this post compared prompt volume with ingest output over the project’s first days. That was useful as an early pressure gauge. It was not an accuracy measure.
A prompt can request a new feature, correct listing data, tune a scraper, change the interface, document a reusable rule, verify completed work, or release code. Counting every one of those against published records made ordinary product development look like AI correction work.
The corrected efficiency measure now counts a prompt once only when it is tagged data_fix or source_ingest. Even that number measures human steering, not record-level error.
Current snapshot
What the project can measure today
Work classification
Most intervention has been data support
The prompt log now assigns each request one primary work type. Release work is separated first. Prompts that touch data_fix or source_ingest then count as data support. Remaining prompts fall into feature development, product quality, or workflow maintenance.
Primary work type
Mutually exclusive totals across 671 logged prompts.
This also changes how code work should be read. Of 327 prompts that changed code, 252 supported data ingestion or repair, 39 supported product-quality fixes, and 26 supported feature development. A code_fix tag says code changed; it does not say the code contained a bug.
Definitions
Workflow tags are not accuracy categories
Source ingest
A source was scraped, normalized, rerun, or reviewed through the ingest pipeline. It is a data-support workflow.
Verification
Tests, API checks, browser checks, ingest reports, duplication reports, or other evidence were recorded. It shows checking, not correctness by itself.
Data fix
Stored or normalized listing, source, organization, venue, or review data changed.
Code fix
Application or pipeline code changed. Its purpose may be data support, feature development, quality, or release work.
Config fix
Source rules, mappings, URLs, categories, or runtime configuration changed.
QA correction
A person found that the result did not match the expected product or source evidence and directed a correction.
A better scorecard
Accuracy has to be measured at several boundaries
| Measure | Current signal | What it answers | What it cannot prove |
|---|---|---|---|
| Pipeline reliability | 516 / 532 runs | Did collection and normalization complete? | A successful run can still contain a semantically wrong field. |
| Rule-check quality | Ingest reports | Are dates, locations, links, types, ages, and required fields structurally usable? | Passing rules does not prove every value matches its source. |
| Human steering load | 55.54 / 1,000 | How much data-focused intervention accompanied published growth? | It is not a record error rate. |
| Verification coverage | 520 prompts | Was explicit checking recorded with the work? | The depth of each check varies. |
| Record-level accuracy | Not yet scored | Does each normalized field match authoritative source evidence? | This requires sampled or comprehensive labeled truth and field-level comparison. |
What the corrections taught us
Repeated misses should change the system
- Stable platforms should use deterministic APIs, feeds, or paging. AI should classify new records or review exceptions, not replace reliable extraction.
- Explicit source fields should beat inference for dates, ages, prices, schedules, links, and listing type.
- Customer-facing fields must never contain parser caveats, uncertainty notes, or implementation commentary.
- Publishing needs separate approval gates. A successful local ingest is not permission to publish locally or push production data.
- When a correction repeats, it should become code, configuration, validation, or a focused test—not another reminder that must be remembered next time.
System design
AI works best inside a controlled pipeline
Instructions remain useful. They preserve judgment and explain edge cases. Code and tests provide enforcement. Reports preserve evidence. Approval gates control state. AI supplies language judgment where the input and output can be constrained.
The most useful accuracy question is not “How many prompts did the AI need?” It is “At each decision boundary, what evidence shows that the result is supported, valid, reviewed, and safe to publish?”
The live Data Quality Report tracks pipeline reliability, validation, field coverage, duplicate flags, and corrections over time. The next meaningful accuracy improvement would be a field-level audit sample tied back to authoritative source evidence.