analyzer_b/README.md

Analyzer B — speed/accuracy benchmarking copy

A working copy of the HVAC blueprint analyzer engine, instrumented for per-step timing and wrapped in a benchmark harness that compares many versions (parallelization and other speed ideas) on a fixed set of confirmed pages — so we can see which version is fastest and which is most accurate.

What's here

File	Purpose
analyze_blueprint_b.py	Copy of python_files/analyze_blueprint.py + timing + page-level parallelism.
hvac_pipeline/	Copied dependency package (schedule extraction, reconcile, etc.).
versions.py	Registry of version configs (the speed ideas). Edit to add up to 10+.
bench.py	Runs versions over the 4 benchmark pages, scores accuracy + speed, writes the leaderboard.
ground_truth/	Human-confirmed unit boxes for the 4 pages (copied from the reviewed runs).
results/benchmark_runs.csv	Append-only log: one row per version+page.
results/LEADERBOARD.md	Auto-generated comparison, sorted fastest + most accurate.
runs/<version>/<bp>/	Per-run output (annotated images, review CSVs, timing_b.json).

Benchmark set (2 blueprints, 4 confirmed pages)

Blueprint	Page	Floor	Confirmed units
355 E 86th	2	Cellar	13
355 E 86th	13	15th	9
953 Dean St	4	Canopy/Entry	25
953 Dean St	6	P/Dn SSDS	66

Engine changes vs the original

1. Per-step timing — wall-clock for schedule / classify / detect written to timing_b.json in the out-dir.
2. Page-level parallelism — HVAC_PAGE_WORKERS (default 1) runs independent plan pages through detection concurrently. Total API concurrency = HVAC_PAGE_WORKERS × HVAC_TILE_WORKERS; keep the product modest to avoid 429s.

Everything else (detection, schedule logic, review CSV format) is unchanged, so accuracy is comparable to the original.

How to run

# one version, cheap validation on the two 355 pages only
.venv\Scripts\python.exe analyzer_b\bench.py --versions v02_tiles4 --quick

# a few versions on all 4 pages
.venv\Scripts\python.exe analyzer_b\bench.py --versions v01_baseline,v02_tiles4,v04_pages2

# the whole sweep
.venv\Scripts\python.exe analyzer_b\bench.py --all

# just regenerate the leaderboard from existing results
.venv\Scripts\python.exe analyzer_b\bench.py --leaderboard-only

Scoring

For each page, predicted boxes are matched to confirmed ground-truth boxes by IoU ≥ 0.3 (boxes at non-300 DPI are rescaled first):

• F1 — spatial detection F1 (headline accuracy).
• Count acc — 1 − |pred − gt| / gt (how close the unit count is).
• Tag acc — fraction of matched boxes whose AC-tag is correct.

Speed is the engine's detect-step seconds (schedule + classify are pre-seeded from cache so every version is compared on the detection work it actually changes).