Synapse - Lease Extractor
source: EDGAR exhibits

Evals · pipeline v0.2

These accuracy numbers are not a quality claim

0 of 5 label files are marked verified. The rest were drafted by the same model being graded, so the accuracy below measures the model agreeing with itself — which is why it reads 100.0%. It is a working harness, not a result.

Verifying the labels is the one task blocking every measurable claim in this project. See corpus/labels/README.md for the procedure and the conventions to settle first.

Field accuracy
100.0%
meaningless — labels unverified
Cost per document
$0.19
$0.93 for a full corpus pass
Latency per document
32s
single extraction call, no OCR
Eval set
5docs
80 labeled fields

Per-field accuracy

Partial matches score half. Uniform bars are the tell: every field scores identically because each label was drafted from the extraction it is grading.

Landlord100%
Tenant100%
Premises100%
Rentable SF100%
Commencement100%
Term length100%
Expiration100%
Base rent100%
Escalations100%
Security deposit100%
Options100%
Notice window100%
Assignment / sublet100%
Early termination100%
Exclusivity100%
CAM / opex100%

Does routing catch the errors?

The question that decides whether confidence scoring earns its keep: of the fields sent to a human, how many were actually wrong — and of the wrong fields, how many did routing catch?

Precision
0.0%
Recall
0.0%

Both read 0% because the labels report zero errors to catch — the tautology again. This number only becomes meaningful once the labels are verified and real disagreements appear.

Label status

The gate on everything above.

  • acm-research-oregon-leasedraft-unverified
  • dexcom-office-leasedraft-unverified
  • hyliion-industrial-leasedraft-unverified
  • tenaya-lab-leasedraft-unverified
  • yoshiharu-retail-leasedraft-unverified

Open judgement calls (7)

Flagged during drafting — each needs a human decision before its label can be marked verified.

Per document

DocumentAccuracyCostLatencyLabels
acm-research-oregon-lease100.0%$0.1420sverified
dexcom-office-lease2 in review100.0%$0.451m 8sverified
hyliion-industrial-lease6 in review100.0%$0.0427sverified
tenaya-lab-lease3 in review100.0%$0.1220sverified
yoshiharu-retail-lease6 in review100.0%$0.1825sverified