# Shipping a Knee Osteo Arthritis pre-read. Numbers, FNs and learnings.

Repo: [github.com/vishsec/subvision](http://github.com/vishsec/subvision).

Demo: [https://subvision-vishsec.streamlit.app](https://subvision-vishsec.streamlit.app/)

---

Every knee-AI demo you've seen leads with its best number. This one leads with its worst case

![](https://cdn.hashnode.com/uploads/covers/65e9c3d6b94261a8cf227d91/44cb7b89-52f6-477c-951d-efcf43c49217.png align="center")

That image is ground-truth KL grade 2: early osteoarthritis. The model says *negative* with 83.6% confidence (P=0.1643 printed on screen both raw and as 16.4%, because a probability nobody can read isn't communication — it was 0.09 before I added flip-averaging at inference; TTA softened the confident miss but did not fix it). It is a false negative, it is on my test set, and the live demo will serve it to you right now.

## The task, deliberately narrow

A **pre-read aid**: single-view knee X-ray in, probability of KL ≥ 2 out, Grad-CAM heat, and a permanent banner *pre-read only, not a diagnosis, requires radiologist sign-off*.

KL (Kellgren-Lawrence) 0-1 = no OA, 2-4 = OA. Binary on purpose: the graded 0-4 middle is where most models underperform.

## Data before model

Rule I kept from my mentor: logic and counts first, model second. Before any training code:

- **train 5,778 / val 826 / test 1,656** (935 negative / 721 positive), OAI-style KL 0-4 folders, counted by hand with a 20-line script.
- Labels are image-level only. **No joint-space-narrowing scores, no osteophyte annotations, nothing about where.** Just yes/no per image.

That last point matters. Whatever the model learned, it learned from a bucket of graded pixels nobody taught it anatomy.

## The numbers (test, n=1,656)

Base model, one pass:

|  |  |
| --- | --- |
| AUC | 0.945 |
| Accuracy | 86.2% (TN 844, FP 91, FN 137, TP 584) |
| Sensitivity | 0.810 |
| Specificity | 0.903 |

**Shipped as v0.2.0 — hflip TTA** (average of the normal pass and a width-mirrored pass; knee anatomy is left-right symmetric):

|  |  |
| --- | --- |
| AUC | **0.950** |
| Accuracy | **86.8%** (TN 852, FP 83, FN 135, TP 586) |
| Sensitivity | 0.813 |
| Specificity | 0.911 |

Is +0.005 AUC real or noise? Paired bootstrap, 10k resamples: test CI **\[+0.0023, +0.0079\]** P≈0.9999, gate set (n=1,526) CI **\[+0.0024, +0.0083\]** P=1.0000 — the gain's confidence interval excludes zero on both splits. Accuracy gains are directional only (P 0.89 / 0.74), so I don't claim those. Controls: vertical flip and 180° rotation *degrade* to AUC 0.884/0.879 — implausible anatomy, the sanity check that keeps the honest flip honest — and hflip without averaging was weaker. Cost: latency ≈2× (local p50 70 → 201 ms, cloud live 201 ms).

Threshold is a **choice**, not a setting I picked once (single-pass sweep; shipped TTA at 0.5 = FN 135 / FP 83):

| thr | FN | FP | read |
| --- | --- | --- | --- |
| 0.3 | 96 | 137 | don't-miss: catches 41 more positives, +46 false alarms |
| **0.5** | **137** | **91** | default, conservative (high spec) |
| 0.7 | 179 | 50 | quiet: half the alarms, 42 more misses |

Accuracy is flat ~0.86 across all three — the threshold moves errors, it doesn't remove them. For a pre-read that queues work to a radiologist, 0.5 is the honest default and the 0.3 option is documented.

## Where the 228 errors live

Per-KL accuracy (single-pass v0; TTA brought the total to 218 errors — the breakdown below is the version I measured per-grade):

```plaintext
KL0  96.6%   KL1  76.7%   KL2  70.9%   KL3  96.9%   KL4  100%
```

**87% of all errors (199/228) sit in KL1 and KL2**, the borderline grades where even humans argue. The extremes are nearly solved; the middle is the whole problem.

Grad-CAM explains why. I pulled 5 true positives and 5 false negatives (all KL2):

- **Hit (p=1.00):** heat sits *on* the lateral joint margin — where the narrowing and osteophytes actually are.
- **Miss (p=0.09):** heat sits *off* the joint — a midline notch blob and a corner artifact. The model got distracted and never looked at the joint space.

**\[SCREENSHOT 2: Grad-CAM pair — fn2_kl2_0.09_9006407R vs tp2_kl2_1.00_9007827R\]**

![](https://cdn.hashnode.com/uploads/covers/65e9c3d6b94261a8cf227d91/92f0b395-2527-4543-86ed-a9a363a1eda2.png align="center")

![](https://cdn.hashnode.com/uploads/covers/65e9c3d6b94261a8cf227d91/e3deb831-878b-45c2-bf0a-a52b9b0524f1.png align="center")

So the fix isn't a threshold — my hypothesis for v1: joint-crop before classification, plus 5-class training. That hypothesis was run (see What's next) and it lost. Keeping the sentence anyway: this is what the evidence said *before* the experiment, which is the only honest way to run one.

Another finding I didn't expect: the model was never taught joint-space narrowing or osteophytes, and it found the gap-and-edge pattern anyway. From binary labels alone.

## What runs behind the demo:

- **ONNX export** (28.2 MB) with a parity check against PyTorch: diff **0.000000**. Serving is torch-free — onnxruntime, numpy, pillow, pydicom. That's it.
- **FastAPI** (`/health`, `/predict`) with PNG/JPG/DICOM auto-detect (DICM magic bytes), DICOM windowing handled, and a latency breakdown in *every* response.
- **tests:** health, PNG predict, **PNG↔DICOM parity on the same image (prob diff < 1e-3)**, and determinism (same input → identical probability, twice).
- **Benchmark, n=50, CPU:** same-day A/B — TTA off p50 **69.8 ms** / on p50 **200.6 ms** total on my laptop. On Streamlit Cloud with TTA: **201.2 ms** (74 decode + 127.2 inference).
- **Calibration:** ECE **0.049** on the gate set (n=1,526), reliability table in repo probabilities are checked, not assumed.
- **Uncertainty band:** p ∈ \[0.2, 0.8\] -> `borderline` -> queue for radiologist sign-off, on API + report + UI. Every response carries `model_version` + `request_id` for audit.
- **Drift check** (new batch vs baseline, PSI) and **GitHub Actions** running the test suite on push.
- **Docker image** (python:3.11-slim) built, run, `/health` verified.
- **Streamlit UI imports the FastAPI module** — one inference function, no second implementation to drift.

Live: [**https://subvision-vishsec.streamlit.app/**](https://subvision-vishsec.streamlit.app/) Repo: [**github.com/vishsec/subvision**](http://github.com/vishsec/subvision)

## What I don't claim and doesn't exist in this version.

No diagnosis. No exact KL grade. No JSN/osteophyte scores. No medial/lateral call. No bilateral read. No fracture or implant rule-out. Single-view, OAI-style data — **needs local validation before any real-world use.** Not a medical device, not for commercial or clinical use.

## The experiment I ran and what it found.

| run | AUC | acc | KL1 | KL2 |
| --- | --- | --- | --- | --- |
| v0 (binary, shipping) | 0.944 | 0.872 | **79.3** | 72.0 |
| 5-class full-film | **0.945** | 0.871 | 75.3 | **74.2** |
| 5-class + joint crop | 0.943 | 0.860 | 79.3 | 66.3 |

**Neither beat v0.** Crop actually hurt KL2, the fixed box + resolution cost threw away the subtle joint-space signal it was supposed to isolate. The 5-class head traded KL2 gains for KL1 losses, netting flat.

So the conclusion is the pre-agreed one: **the middle grades are label-limited, not epochs-limited.** No retrain won. Production stays v0.

I'm keeping this section because "I tried the obvious fix, it failed, here's the table" is the part most ML posts delete.

## What this is limited by

The ceiling here is not compute it's labels.

- **Labels are image-level only.** One KL number per film: no joint-space-narrowing scores, no osteophyte marks, no boxes the model was never taught anatomy. 87% of all errors sit in KL1/KL2, the exact grades where radiologists argue. Both retrains I ran (5-class, joint-crop) failed to beat v0, so the bottleneck is label quality and quantity, not epochs. The fix on the list. a joint box set or a labeling review with a radiologist.
- **Labelled images are a budget item, not a download.** ~8.8k graded films exist here (OAI-style, single-view). Scaling labels means either paying a radiologist per film or finding a cohort behind an institutional signature the same wall that stopped external validation. Until that wall falls, every number stays in-cohort.
- **~1 in 5 positives still missed** at the default threshold (sens 0.813 → 135 FN on test). That's the price of a conservative pre-read; the 0.3 don't-miss option is documented, but the misses don't disappear, they just move.
- **Probability ≠ clinical risk.** ECE 0.049 is measured, not assumed but a softmax score is still not a calibrated disease probability for one patient.
- **Resources:** trained on a free Kaggle GPU, served on CPU with a 28 MB ONNX file no fine-tuned foundation model.