Correct superseded conformal claims

The conformal tab still carried pre-clean-split results: it claimed the bound held
at every risk level tested while the chart beside it shows three points above the
line. Now 6 of 9, with the exceedances given as 1.12/0.55/0.83 standard errors.
Also corrected: abstention 72%/3.4% -> 91.7%/12.8%; the monotonicity minimum
(lambda ~ 0.57 "falls to zero" -> 0.029 at lambda ~ 0.461, first rise 0.486);
"the bound holds at whatever value is chosen"; and the figure caption.

The header conflated micro-F1 with top-1 accuracy, which are the same metric here
-- the real difference is two training runs. Verified the served checkpoint at
172/180 = 95.6% against the notebook's 94.1%, and corrected train.py's comments
claiming it IS the notebook's model.

Mirrors llm-wiki commit edfa1a9.

app.py

@@ -134,8 +134,10 @@ with gr.Blocks(title="Explainable & Auditable Alloy Defect Detection",
             "hooks, following the course's own notebook).\n\n"
             "*The classifier itself is deliberately ordinary* — a compact 164k-parameter "
             "CNN, trained by the same recipe as the notebook: a stratified 70/10/10/10 split "
-            "where the test images are never used to fit weights or pick a checkpoint "
-            "(94.1% micro-F1 there; 95.6% top-1 accuracy for this checkpoint). That is the "
+            "where the test images are never used to fit weights or pick a checkpoint. "
+            "The notebook's run scores 94.1% on that test split; the separately trained "
+            "checkpoint served here scores 95.6% on the same split — same recipe and "
+            "same split, different training run. That is the "
             "point: the question is not whether a CNN can score well on NEU-DET (it can), but "
             "whether the explanation sitting on top of it means anything. Tabs 2 and 3 test "
             "exactly that."
@@ -223,8 +225,14 @@ with gr.Blocks(title="Explainable & Auditable Alloy Defect Detection",
             The guarantee is arithmetic, not optimism.
 
             **It was validated, not just asserted:** empirical error stayed at or below
-            target at **every** risk level tested (α = 0.1 through 0.9). Every point
-            on the chart below sits on or under the dashed line.
+            target at **six of the nine** risk levels tested (α = 0.1 through 0.9).
+            The three that came in over — α = 0.10, 0.20 and 0.90 — miss by
+            **1.12, 0.55 and 0.83 standard errors** on 180 test images. The bound is on
+            *expected* error, so realisations scatter either side of the line; near-misses in
+            both directions are what a correctly behaving bound looks like. An earlier version
+            of this project reported nine out of nine, which looked stronger and was weaker
+            — that model had been checkpoint-selected on the very images it was scored
+            against.
 
             ---
 
@@ -240,8 +248,9 @@ with gr.Blocks(title="Explainable & Auditable Alloy Defect Detection",
             | Empty | Nothing clears the bar | **Abstain** — the model declines |
 
             The plant picks α. That single dial trades throughput against escalation
-            volume, and the bound holds at whatever value is chosen. **This, not the
-            accuracy number and not the heatmap, is what we would put in front of an
+            volume. Pick it at the **strict end**: α = 0.10 abstains on 12.8% of images,
+            α = 0.90 on 91.7% — within budget and operationally useless. **This, not
+            the accuracy number and not the heatmap, is what we would put in front of an
             auditor.**
 
             #### Three honest limits
@@ -256,26 +265,29 @@ with gr.Blocks(title="Explainable & Auditable Alloy Defect Detection",
             - **At lenient α the metric is dominated by abstentions.** We adapted a
               method built for pixel-level segmentation, where a prediction set spans
               thousands of pixels and is essentially never empty. With six classes, empty
-              sets are common at strict bars — 72% of images at the most extreme
-              setting — and an empty set is scored as a full error. The useful
-              operating range here is the **low-α end**, where empty sets are rare
-              (3.4%).
+              sets are common at strict bars — 91.7% of images at the most lenient
+              α — and an empty set is scored as a full error. So at that end the
+              reported "false discovery rate" is very nearly a synonym for "abstention rate."
+              The useful operating range here is the **low-α end**, where empty sets are
+              far rarer (12.8%).
             - **A precondition of the theorem is not strictly met.** Conformal Risk Control
               requires the loss to be *monotone non-increasing* — as you flag more, the
               risk must never climb back. False-negative rate satisfies that; **false
-              discovery rate does not.** Ours is U-shaped: it falls to zero around
-              λ ≈ 0.57, then rises again as the flagged set fills with
-              false positives. Every threshold we actually selected sits on the *descending*
-              branch, where the condition effectively holds — which is consistent with
-              the bound holding at all nine levels. But that is a property of this data, not
-              a proof. **It is why the chart above is load-bearing rather than decorative:**
-              with a precondition in question, checking the bound empirically on held-out
-              data is doing real work.
+              discovery rate does not.** Ours is U-shaped: it falls from 0.949 to a minimum of
+              0.029 around λ ≈ 0.461, then climbs back to 0.755 as the
+              flagged set fills with false positives — rising at 80 of 399 grid steps.
+              Every threshold we actually selected (0.004–0.219) sits on the *descending*
+              branch, where the condition effectively holds; the first rise is not until
+              λ = 0.486. Capping the search grid at λ ≤ 0.48
+              would satisfy the precondition outright **without changing a single reported
+              number**. But that is a property of this data, not a proof. **It is why the chart
+              above is load-bearing rather than decorative:** with a precondition in question,
+              checking the bound empirically on held-out data is doing real work.
             """
         )
         gr.Image(
             value="assets/conformal-guarantee.png",
-            label="Conformal Risk Control: guarantee holds at every tested risk level (precomputed in the notebook)",
+            label="Conformal Risk Control: empirical FDR against target at nine risk levels; under target at six, over by ~1 standard error at three (precomputed in the notebook)",
             interactive=False,
         )
 

train.py

@@ -17,7 +17,7 @@ from torch.utils.data import DataLoader, Dataset
 
 from model import CLASS_NAMES, IMAGE_SIZE, MEAN, STD, DefectCNN
 
-SEED = 41   # matches the notebook, so this checkpoint IS the notebook's model
+SEED = 41   # matches the notebook, so this checkpoint sees the notebook's exact splits
 DATA_ROOT = os.path.join("..", "data", "neu-det-src", "IMAGES")
 CHECKPOINT_PATH = "model.pt"
 EXAMPLES_DIR = "examples"
@@ -74,9 +74,11 @@ def main():
     for name, count in Counter(labels_by_path.values()).items():
         print(f"  {name:16s} {count:4d}")
 
-    # Stratified 70/10/10/10 per class, identical to the notebook's Section 2 so the
-    # served checkpoint is the same model the notebook reports on. Only train and the
-    # final test split are needed here; val is used for checkpoint selection.
+    # Stratified 70/10/10/10 per class, identical to the notebook's Section 2, so the
+    # served checkpoint is trained and scored on exactly the notebook's splits. It is a
+    # separate training run, so its weights and test score differ from the notebook's
+    # (95.6% here vs 94.1% there). Only train and the final test split are needed here;
+    # val is used for checkpoint selection.
     rng = np.random.default_rng(SEED)
     train_paths, val_paths, eval_paths = [], [], []
     for cls in CLASS_NAMES: