# HVAC Blueprint Analyzer: Research Paper Outline

## Working Title
**Automated HVAC Schedule Extraction from Mechanical Blueprints using Multi-Modal OCR and Vision-Language Models**

*Or: "Bridging Legacy Blueprints and Modern Facilities Data: A Tiered Extraction Pipeline for HVAC Equipment Schedules"*

---

## Metadata

- **Authors:** [Your name]
- **Affiliation:** [Your org/university]
- **Target Venues:** 
  - ICDAR (International Conference on Document Analysis and Recognition) — primary focus on OCR/document analysis
  - CVPR/ICCV (Computer Vision) — secondary if emphasizing vision/detection aspects
  - IEEE/ACM journals — Construction/facilities management focus
- **Status:** Outline / In Progress
- **Last Updated:** June 2026

---

## Abstract (150-250 words)

**Problem Statement:**
Mechanical HVAC blueprints are dense, rasterized documents with no selectable text. Extracting equipment schedules and locating units on floor plans remains a manual task, limiting scalability in large facilities projects.

**Approach:**
This paper presents a tiered extraction pipeline that combines deterministic methods (pdfplumber, Tesseract OCR) with vision-language models (Google Gemini) to achieve robust schedule extraction without manual annotation. The pipeline classifies pages, extracts equipment schedules using multi-tier fallback logic, and trains YOLO detectors to localize unit tags on floor plans.

**Contributions:**
1. First systematic approach to automated HVAC schedule extraction from blueprints
2. Confidence-based tiered pipeline balancing accuracy, cost, and latency
3. Open-source implementation with extensible architecture
4. Evaluation on [N] diverse blueprints showing [X]% schedule extraction accuracy and [Y]% unit localization precision

**Keywords:** Document analysis, OCR, Vision-language models, HVAC, Blueprint processing, Information extraction

---

## 1. Introduction (2-3 pages)

### 1.1 Motivation & Problem Context

**[Subsection content to write]**

- **Industry Challenge:** Mechanical contractors, architects, and facilities managers must manually cross-reference blueprints to:
  - Identify all HVAC units in a building
  - Match unit tags to equipment specs
  - Locate units on floor plans for installation/replacement
  - Audit system consistency across floors
  - Plan maintenance schedules

- **Scale Problem:** A 20-floor office building might have 200+ indoor HVAC units across 50+ blueprint pages. Manual extraction takes 8-16 hours per project.

- **Technical Challenge:** Blueprints are typically:
  - Rasterized PDFs (no selectable text)
  - High variability in layout/quality across architects
  - Dense with technical symbols and annotations
  - Scanned at varying resolutions and compression
  - Inconsistent in HVAC naming conventions

### 1.2 Why Existing Solutions Fall Short

**[Subsection content to write]**

- **Manual Extraction** — Slow, error-prone, doesn't scale
- **BIM/IFC Approaches** — Require digitally-native CAD models; don't work on legacy blueprints (many buildings pre-date digital standards)
- **Generic OCR (Tesseract-only)** — Fails on poor-quality rasterized pages; low accuracy on mixed text/graphics
- **Generic Vision Models (Gemini-only)** — Can work but expensive per page; no iterative refinement; hard to debug
- **No unified pipeline** — Prior work addresses pieces (table extraction, floor plan detection) but not end-to-end schedule + location extraction

### 1.3 This Paper's Approach

**[Subsection content to write]**

- Tiered pipeline: local methods → cloud VLM fallback
- Confidence-based routing (don't escalate if local methods high-confidence)
- Explicit data reconciliation (equipment reference sheet lookup)
- Floor plan unit localization using YOLO
- Extensible architecture for other technical documents

### 1.4 Contributions

**[Subsection content to write]**

1. **Problem Formalization** — Defines blueprint schedule extraction as a structured extraction + localization problem
2. **Methodological Framework** — Tiered pipeline with confidence gates; shows how to combine OCR, Tesseract, and VLMs
3. **HVAC Domain Application** — First work on blueprint-based HVAC extraction
4. **Practical Implementation** — Open-source tool; addresses real-world messy data
5. **Empirical Evaluation** — Quantitative results on diverse blueprints; ablation study

### 1.5 Paper Organization

Brief outline of sections.

---

## 2. Related Work (2-3 pages)

### 2.1 Document Analysis & OCR

**[Subsection content to write — pull from RESEARCH_FOUNDATION.md]**

- Overview of document digitization challenges
- Tesseract OCR: strengths, limitations on engineering drawings
- Modern deep learning approaches (DeepSeek-OCR, etc.)
- Why tiered approaches are justified

### 2.2 Vision-Language Models for Document Understanding

**[Subsection content to write]**

- Shift from OCR-only to multimodal (text + vision)
- Gemini, GPT-4V, LLaVA capabilities for document analysis
- Benefits: robustness to layout variation, mixed-media documents
- Limitations: API cost, latency, interpretability
- Your tiered approach: use VLM strategically, not everywhere

### 2.3 Table & Structured Data Extraction

**[Subsection content to write]**

- pdfplumber overview and performance (96% recognition)
- Limitations on complex/rasterized tables
- Transformer-based approaches (UniTable, etc.)
- Your choice: deterministic + VLM fallback

### 2.4 Object Detection on Architectural Plans

**[Subsection content to write]**

- YOLO for floor plan analysis
- YOLOv5 vs. YOLOv8 on floor plan datasets
- Room/wall/door detection prior work
- Unit tag localization (your novel task)

### 2.5 HVAC & Building Systems Data

**[Subsection content to write]**

- BIM/IFC-based HVAC extraction (doesn't apply to legacy blueprints)
- Building energy modeling needs
- Facilities management data requirements
- Why blueprint-based extraction fills a gap

### 2.6 Document Image Classification

**[Subsection content to write]**

- Page type classification (schedule vs. floor plan)
- Layout-based heuristics vs. deep learning
- Your approach: simple heuristics + VLM fallback for edge cases

---

## 3. Problem Formulation & Data Model

### 3.1 Input & Output Specification

**[Subsection content to write]**

**Input:**
- Mechanical HVAC blueprint PDF (rasterized or mixed)
- Variable: 10-100 pages, resolution 200-600 DPI, quality ranging from clear to degraded scans

**Output - Schedule:**
```
{
  tag: str,           # "AC-A", "AC-LV", "AC-C-1", etc.
  quantity: int,      # Number of units with this tag
  unit_type: str,     # "Wall Mounted", "Ceiling Cassette", etc.
  model: str,         # Full model number, e.g., "FXAQ09PVJU"
  brand: str,         # "Mitsubishi", "Daikin", etc.
  confidence: float   # 0-1, confidence in this extraction
}
```

**Output - Floor Plan Annotations:**
- Per-floor image with bounding boxes around detected unit tag labels
- YOLO label format: class, x_center, y_center, width, height (normalized)

### 3.2 Supported Tag Formats

**[Subsection content to write]**

Document the regex/parsing rules for tag formats:
- Single letter: `AC-A`, `AC-B`, ..., `AC-U`
- Multi-letter: `AC-LV`, `AC-R`
- Letter + number: `AC-C-1`, `AC-C-2`, ..., `AC-C-11`
- Number + number: `AC-1-1`, `AC-1-2`, ..., `AC-1-14`

### 3.3 Data Quality Challenges

**[Subsection content to write]**

- Inconsistent nomenclature across architects
- Abbreviated unit types (need reference sheet lookup)
- Typos/OCR errors in model numbers
- Missing unit types (must infer from model prefix)
- Quantity encoding variations (qty column header name varies)

---

## 4. Methodology

### 4.1 System Architecture Overview

**[Subsection content to write + diagram]**

```
Input: PDF
  ↓
[Page Classification] → Classify pages (schedule vs. floor plan)
  ↓
[Schedule Extraction Tier 1] → pdfplumber table extraction
  ↓ if low-confidence
[Schedule Extraction Tier 2] → pdfplumber text extraction (spatial)
  ↓ if low-confidence
[Schedule Extraction Tier 3] → Tesseract OCR (PSM modes)
  ↓ if low-confidence or incomplete
[Schedule Extraction Tier 4] → Gemini 2.5 Flash vision
  ↓
[Data Reconciliation] → Lookup unit types, validate completeness
  ↓
[Floor Plan Detection] → Render pages, run YOLO detector
  ↓
Output: schedule.json, annotated_floor_plans/
```

### 4.2 Tier 1: Native PDF Extraction (pdfplumber)

**[Subsection content to write]**

- **Method:** pdfplumber table detection + text extraction
- **When Used:** All PDFs, first attempt
- **Confidence Metric:** # of tags found, # of complete records
- **Fallback Threshold:** < 5 tags OR any tag missing unit_type → escalate
- **Strengths:** Deterministic, fast, no API calls
- **Limitations:** Fails on rasterized/poorly formatted tables

### 4.3 Tier 2: Spatial Text Recovery

**[Subsection content to write]**

- **Method:** Extract all text from schedule page; parse via regex + heuristics
- **When Used:** Tier 1 failed
- **Parsing Logic:** Document the regex patterns and positional logic
- **Confidence Metric:** Regex match strength, spatial coherence
- **Fallback Threshold:** < 50% confidence → escalate

### 4.4 Tier 3: Tesseract OCR

**[Subsection content to write]**

- **Method:** Render page at 250 DPI grayscale; run Tesseract with PSM 4 and PSM 6
- **PSM Choices:**
  - PSM 4: Assume a single column of text of variable sizes
  - PSM 6: Assume a single uniform block of text
- **When Used:** Tier 2 failed
- **Confidence Metric:** Tesseract confidence scores, regex match success
- **Fallback Threshold:** < 40% average confidence → escalate
- **Strengths:** Works on many rasterized pages; fully local (no API)
- **Limitations:** Struggles with poor quality, mixed graphics

### 4.5 Tier 4: Vision-Language Model Fallback (Gemini)

**[Subsection content to write]**

- **Model:** Google Gemini 2.5 Flash
- **Prompt:** Structured extraction request (return as CSV)
- **When Used:** Tier 3 failed or low-confidence
- **Input:** Full schedule page image
- **Output Format:** CSV-like string parsed into schedule records
- **Cost:** ~0.15 API credits per page (empirical measurement)
- **Latency:** ~2-5 seconds per page
- **Fallback Threshold:** N/A (final tier)
- **Strengths:** Robust to layout/quality variation; understands context
- **Limitations:** Expensive; slower; less interpretable; prone to hallucination on unfamiliar formats

### 4.6 Data Reconciliation: Unit Type Lookup

**[Subsection content to write]**

- **Problem:** Some tags extracted without unit_type (abbreviated or corrupted OCR)
- **Solution:** Model number prefix matching against reference sheet
  - Reference sheet: `Mitsubishi_Daikin_Indoor_HVAC_Units_v1.xlsx`
  - Longest-prefix matching: `FXAQ09PVJU` → prefix `FXAQ` → "Wall Mounted"
- **Coverage:** ~95% of model prefixes; unmatchable models flagged for manual review

### 4.7 Page Classification

**[Subsection content to write]**

- **Task:** Classify each page into {schedule, floor_plan, detail, notes}
- **Method:** Heuristics + Gemini fallback
  - Heuristics: detect table presence, text density, graphics density
  - If heuristics ambiguous → ask Gemini to classify
- **Why Important:** Routes pages to specialized extraction logic; floor plans → YOLO detection

### 4.8 Floor Plan Unit Localization via YOLO

**[Subsection content to write]**

- **Task:** Find unit tag labels on floor plan pages; generate bounding boxes
- **Method:** 
  1. Render floor plan at high DPI (300-600)
  2. Crop patches around text regions (OCR-detected tag candidates)
  3. Train YOLO to detect tag bounding boxes
  4. Generate YOLO label format (class=unit_tag, bbox_norm_coords)
- **Training Data:** Annotated floor plan images
- **Model:** YOLOv8 (current best-performing on floor plan datasets)
- **Inference:** Real-time on page-sized images

### 4.9 Implementation Details

**[Subsection content to write]**

- **Language/Stack:** Python 3.10+
- **Key Libraries:**
  - `pdfplumber` (table/text extraction)
  - `pytesseract` + Tesseract (OCR)
  - `opencv-python` (image processing)
  - `google-generativeai` (Gemini API)
  - `yolov8` (object detection)
  - `pandas` (data wrangling)
- **Caching:** Schedule extraction cached as JSON (avoid re-running Gemini)
- **Configuration:** Command-line flags for DPI, PSM, thresholds, etc.

---

## 5. Evaluation

### 5.1 Dataset

**[Subsection content to write]**

- **Source:** Mechanical blueprints from [N buildings / projects]
- **Size:** [N] PDFs, [M] pages, [K] unique unit tags
- **Variation:** Architects, eras, quality levels, naming conventions
- **Ground Truth:** Manual annotation by [you / domain expert]
- **Splits:** Train / validation / test (if applicable)

**Table: Dataset Statistics**
```
| Metric | Value |
|--------|-------|
| # PDFs | [?] |
| # Pages | [?] |
| # Unit tags | [?] |
| # Floor plan pages | [?] |
| Avg. quality (1-5) | [?] |
| Architect diversity | [?] |
```

### 5.2 Metrics

**[Subsection content to write]**

**Schedule Extraction:**
- **Precision:** # correctly extracted records / # records extracted
- **Recall:** # correctly extracted / # true records (ground truth)
- **F1-score:** Harmonic mean
- **Per-field accuracy:** Separate metrics for tag, quantity, unit_type, model, brand
- **Confidence Calibration:** Does predicted confidence match actual accuracy?

**Floor Plan Detection:**
- **Precision:** # correct detections / # detections
- **Recall:** # correct detections / # ground truth tags
- **mAP (mean Average Precision):** Standard YOLO evaluation metric

**System-Level:**
- **End-to-End Accuracy:** % of schedules fully extracted without manual intervention
- **Cost per Blueprint:** Average API calls + execution time
- **Latency:** Total wall-clock time per PDF

### 5.3 Baselines

**[Subsection content to write]**

1. **Tesseract Only** — Tier 3 alone, no fallback
2. **Gemini Only** — Direct vision extraction without local tiers
3. **Manual Extraction** — Human baseline (time + cost)
4. **Simple OCR + Regex** — Baseline from prior work (if available)

### 5.4 Ablation Study

**[Subsection content to write]**

- **Remove Tier 1 (pdfplumber):** What's the cost of skipping native extraction?
- **Remove Tier 2 (spatial text):** Impact of skipping heuristic parsing?
- **Remove Tier 3 (Tesseract):** Does OCR add value over VLM alone?
- **Remove Tier 4 (Gemini):** What's the ceiling of local methods?
- **Impact of Confidence Thresholds:** Sensitivity analysis on tier fallback triggers

### 5.5 Statistical Significance

**[Subsection content to write]**

- Confidence intervals on reported metrics
- Cross-validation if dataset size permits
- Significance tests comparing to baselines

---

## 6. Results

### 6.1 Overall Performance

**[Subsection content to write + tables/charts]**

**Table: Schedule Extraction Results**
```
| Method | Precision | Recall | F1-Score |
|--------|-----------|--------|----------|
| Tesseract Only | [?] | [?] | [?] |
| Gemini Only | [?] | [?] | [?] |
| Tiered (Ours) | [?] | [?] | [?] |
```

**Table: Per-Field Accuracy**
```
| Field | Accuracy |
|-------|----------|
| Tag | [?]% |
| Quantity | [?]% |
| Unit Type | [?]% |
| Model | [?]% |
| Brand | [?]% |
```

**Table: Floor Plan Detection (YOLO)**
```
| Metric | Value |
|--------|-------|
| mAP (IOU=0.5) | [?] |
| Precision | [?] |
| Recall | [?] |
```

### 6.2 Tier Performance Breakdown

**[Subsection content to write + charts]**

- % of PDFs fully resolved at Tier 1, 2, 3, 4
- Distribution of fallback triggers
- Visualization: stacked bar chart showing tier utilization

**Chart: Tier Utilization**
```
Tier 1 (pdfplumber): ??% of PDFs
Tier 2 (spatial): ??% of remaining
Tier 3 (Tesseract): ??% of remaining
Tier 4 (Gemini): ??% of remaining
```

### 6.3 Cost Analysis

**[Subsection content to write]**

- **API Costs:** Average $ per PDF (Gemini calls)
- **Latency:** Wall-clock time per PDF (segmented by tier)
- **Tradeoff:** Cost vs. accuracy at different confidence thresholds

**Table: Cost-Accuracy Tradeoff**
```
| Tier Strategy | API Calls/PDF | Accuracy | Cost/Accuracy |
|---------------|---------------|----------|---------------|
| Gemini-only | 1 | [?]% | [?] |
| Tiered (current) | [?] | [?]% | [?] |
| Aggressive Tier 4 | [?] | [?]% | [?] |
```

### 6.4 Failure Analysis

**[Subsection content to write]**

- Common failure modes (e.g., abbreviated units, poor OCR on specific fonts)
- Case studies: 3-5 examples of difficult blueprints
- Systematic errors (e.g., "AC-1" misread as "AC-I")
- What confidence thresholds miss

### 6.5 User Study (Optional)

**[Subsection content to write if applicable]**

- Domain expert validates extracted schedules
- % agreement with manual annotation
- Time savings vs. manual extraction

---

## 7. Discussion

### 7.1 Why the Tiered Approach Works

**[Subsection content to write]**

- Deterministic tiers (Tiers 1-3) handle majority of well-formed PDFs
- Gemini fallback handles edge cases without polluting cost
- Confidence gates prevent false confidence propagation
- Separates concerns: layout parsing, OCR, semantic understanding

### 7.2 Limitations

**[Subsection content to write]**

1. **Limited to HVAC systems** — Would need retraining/tuning for other HVAC brands or other mechanical equipment
2. **Assumes standard naming conventions** — Custom tag formats require regex updates
3. **Gemini API dependency** — Cost scales with poorly-digitized blueprints; T4 requires internet
4. **YOLO requires labeled floor plans** — Unit localization task needs per-building annotation
5. **No handling of non-English blueprints** — OCR/VLM may struggle with non-Latin scripts
6. **Manual validation still needed** — Extracted data should be spot-checked by domain expert

### 7.3 Broader Implications

**[Subsection content to write]**

- **Generalizability:** Could this approach work for:
  - Electrical plans → breaker schedules, device locations?
  - Plumbing plans → fixture schedules, valve locations?
  - Structural drawings → column/beam tagging?
- **Legacy Document Digitization:** How much of the construction industry still relies on rasterized blueprints? (Large addressable problem)
- **Hybrid Human-AI Workflows:** Extraction as a semi-automated augmentation, not replacement, for domain experts

### 7.4 Future Work

**[Subsection content to write]**

1. **Dataset Release** — Publish annotated blueprint dataset (if IP/privacy permits) to enable reproduction and future work
2. **Multi-Brand Support** — Extend reference sheet; test on Lennox, Carrier, York, etc.
3. **Intelligent Tier Selection** — Learn which tier to skip for a given blueprint based on visual features
4. **Fine-Tuned VLM** — Train custom Gemini or Llama model on blueprint domain; compare to generic Gemini
5. **Real-Time Interactive Feedback** — Allow user corrections mid-extraction; update extraction logic
6. **Cross-Page Reconciliation** — Detect inconsistencies in unit tags across pages; flag for manual review
7. **Building-Wide HVAC Graph** — Link floor plan locations to schedule data; visualize system topology

---

## 8. Conclusion

**[Subsection content to write]**

- **Problem:** Manual extraction of HVAC schedules from blueprints is slow, error-prone, and doesn't scale
- **Solution:** Tiered extraction pipeline (local deterministic methods → VLM fallback) balances accuracy, cost, and interpretability
- **Results:** [X]% schedule extraction accuracy, [Y]% unit localization precision on [N] diverse blueprints
- **Impact:** [Hours/cost saved per project]; applicable to broader document digitization in construction
- **Contribution:** First systematic approach to automated HVAC blueprint extraction; open-source implementation; extensible framework

---

## Appendices (Optional)

### A. Command-Line Interface Examples

```bash
# Full run with auto-confirm
python analyze_blueprint.py "raw_pdfs/my_project.pdf" --yes

# Schedule extraction only
python analyze_blueprint.py "raw_pdfs/my_project.pdf" --step schedule

# Custom DPI and Tesseract PSM
python analyze_blueprint.py "raw_pdfs/my_project.pdf" --dpi 300 --psm 6
```

### B. Sample Output

```json
{
  "tag": "AC-A",
  "quantity": 2,
  "unit_type": "Wall Mounted",
  "model": "FXAQ09PVJU",
  "brand": "Mitsubishi",
  "source_tier": 1,
  "confidence": 0.95
}
```

### C. YOLO Label Format

```
0 0.50 0.50 0.20 0.15
```
(class_id, x_center_norm, y_center_norm, width_norm, height_norm)

### D. Reference Sheet Sample

| Prefix | Unit Type | Brand |
|--------|-----------|-------|
| FXAQ | Wall Mounted | Mitsubishi |
| FXSQ | Ceiling Concealed Ducted | Mitsubishi |
| MSZ | Wall Mounted | Mitsubishi |
| PLFY | Ceiling Cassette 4-Way | Daikin |

---

## Author Notes

- **Completion Target:** [Month/Year]
- **Target Page Count:** 8-12 pages (IEEE format) or 15-20 pages (journal submission)
- **Review Cycle:** [Internal review plan]
- **Open Source Release:** [Planned?]