Shipping a Knee Osteo Arthritis pre-read. Numbers, FNs and learnings.

Repo: github.com/vishsec/subvision.

Demo: https://subvision-vishsec.streamlit.app

---

Every knee-AI demo you've seen leads with its best number. This one leads with its worst case

That image is ground-truth KL grade 2: early osteoarthritis. The model says negative with 83.6% confidence (P=0.1643 printed on screen both raw and as 16.4%, because a probability nobody can read isn't communication — it was 0.09 before I added flip-averaging at inference; TTA softened the confident miss but did not fix it). It is a false negative, it is on my test set, and the live demo will serve it to you right now.

The task, deliberately narrow

A pre-read aid: single-view knee X-ray in, probability of KL ≥ 2 out, Grad-CAM heat, and a permanent banner pre-read only, not a diagnosis, requires radiologist sign-off.

KL (Kellgren-Lawrence) 0-1 = no OA, 2-4 = OA. Binary on purpose: the graded 0-4 middle is where most models underperform.

Data before model

Rule I kept from my mentor: logic and counts first, model second. Before any training code:

• train 5,778 / val 826 / test 1,656 (935 negative / 721 positive), OAI-style KL 0-4 folders, counted by hand with a 20-line script.
• Labels are image-level only. No joint-space-narrowing scores, no osteophyte annotations, nothing about where. Just yes/no per image.

That last point matters. Whatever the model learned, it learned from a bucket of graded pixels nobody taught it anatomy.

The numbers (test, n=1,656)

Base model, one pass:

AUC	0.945
Accuracy	86.2% (TN 844, FP 91, FN 137, TP 584)
Sensitivity	0.810
Specificity	0.903

Shipped as v0.2.0 — hflip TTA (average of the normal pass and a width-mirrored pass; knee anatomy is left-right symmetric):

AUC	0.950
Accuracy	86.8% (TN 852, FP 83, FN 135, TP 586)
Sensitivity	0.813
Specificity	0.911

Is +0.005 AUC real or noise? Paired bootstrap, 10k resamples: test CI [+0.0023, +0.0079] P≈0.9999, gate set (n=1,526) CI [+0.0024, +0.0083] P=1.0000 — the gain's confidence interval excludes zero on both splits. Accuracy gains are directional only (P 0.89 / 0.74), so I don't claim those. Controls: vertical flip and 180° rotation degrade to AUC 0.884/0.879 — implausible anatomy, the sanity check that keeps the honest flip honest — and hflip without averaging was weaker. Cost: latency ≈2× (local p50 70 → 201 ms, cloud live 201 ms).

Threshold is a choice, not a setting I picked once (single-pass sweep; shipped TTA at 0.5 = FN 135 / FP 83):

thr	FN	FP	read
0.3	96	137	don't-miss: catches 41 more positives, +46 false alarms
0.5	137	91	default, conservative (high spec)
0.7	179	50	quiet: half the alarms, 42 more misses

Accuracy is flat ~0.86 across all three — the threshold moves errors, it doesn't remove them. For a pre-read that queues work to a radiologist, 0.5 is the honest default and the 0.3 option is documented.

Where the 228 errors live

Per-KL accuracy (single-pass v0; TTA brought the total to 218 errors — the breakdown below is the version I measured per-grade):

KL0  96.6%   KL1  76.7%   KL2  70.9%   KL3  96.9%   KL4  100%

87% of all errors (199/228) sit in KL1 and KL2, the borderline grades where even humans argue. The extremes are nearly solved; the middle is the whole problem.

Grad-CAM explains why. I pulled 5 true positives and 5 false negatives (all KL2):

• Hit (p=1.00): heat sits on the lateral joint margin — where the narrowing and osteophytes actually are.
• Miss (p=0.09): heat sits off the joint — a midline notch blob and a corner artifact. The model got distracted and never looked at the joint space.

[SCREENSHOT 2: Grad-CAM pair — fn2_kl2_0.09_9006407R vs tp2_kl2_1.00_9007827R]

So the fix isn't a threshold — my hypothesis for v1: joint-crop before classification, plus 5-class training. That hypothesis was run (see What's next) and it lost. Keeping the sentence anyway: this is what the evidence said before the experiment, which is the only honest way to run one.

Another finding I didn't expect: the model was never taught joint-space narrowing or osteophytes, and it found the gap-and-edge pattern anyway. From binary labels alone.

What runs behind the demo:

• ONNX export (28.2 MB) with a parity check against PyTorch: diff 0.000000. Serving is torch-free — onnxruntime, numpy, pillow, pydicom. That's it.
• FastAPI (/health, /predict) with PNG/JPG/DICOM auto-detect (DICM magic bytes), DICOM windowing handled, and a latency breakdown in every response.
• tests: health, PNG predict, PNG↔DICOM parity on the same image (prob diff < 1e-3), and determinism (same input → identical probability, twice).
• Benchmark, n=50, CPU: same-day A/B — TTA off p50 69.8 ms / on p50 200.6 ms total on my laptop. On Streamlit Cloud with TTA: 201.2 ms (74 decode + 127.2 inference).
• Calibration: ECE 0.049 on the gate set (n=1,526), reliability table in repo probabilities are checked, not assumed.
• Uncertainty band: p ∈ [0.2, 0.8] -> borderline -> queue for radiologist sign-off, on API + report + UI. Every response carries model_version + request_id for audit.
• Drift check (new batch vs baseline, PSI) and GitHub Actions running the test suite on push.
• Docker image (python:3.11-slim) built, run, /health verified.
• Streamlit UI imports the FastAPI module — one inference function, no second implementation to drift.

Live: https://subvision-vishsec.streamlit.app/ Repo: github.com/vishsec/subvision

What I don't claim and doesn't exist in this version.

No diagnosis. No exact KL grade. No JSN/osteophyte scores. No medial/lateral call. No bilateral read. No fracture or implant rule-out. Single-view, OAI-style data — needs local validation before any real-world use. Not a medical device, not for commercial or clinical use.

The experiment I ran and what it found.

run	AUC	acc	KL1	KL2
v0 (binary, shipping)	0.944	0.872	79.3	72.0
5-class full-film	0.945	0.871	75.3	74.2
5-class + joint crop	0.943	0.860	79.3	66.3

Neither beat v0. Crop actually hurt KL2, the fixed box + resolution cost threw away the subtle joint-space signal it was supposed to isolate. The 5-class head traded KL2 gains for KL1 losses, netting flat.

So the conclusion is the pre-agreed one: the middle grades are label-limited, not epochs-limited. No retrain won. Production stays v0.

I'm keeping this section because "I tried the obvious fix, it failed, here's the table" is the part most ML posts delete.

What this is limited by

The ceiling here is not compute it's labels.

• Labels are image-level only. One KL number per film: no joint-space-narrowing scores, no osteophyte marks, no boxes the model was never taught anatomy. 87% of all errors sit in KL1/KL2, the exact grades where radiologists argue. Both retrains I ran (5-class, joint-crop) failed to beat v0, so the bottleneck is label quality and quantity, not epochs. The fix on the list. a joint box set or a labeling review with a radiologist.
• Labelled images are a budget item, not a download. ~8.8k graded films exist here (OAI-style, single-view). Scaling labels means either paying a radiologist per film or finding a cohort behind an institutional signature the same wall that stopped external validation. Until that wall falls, every number stays in-cohort.
• ~1 in 5 positives still missed at the default threshold (sens 0.813 → 135 FN on test). That's the price of a conservative pre-read; the 0.3 don't-miss option is documented, but the misses don't disappear, they just move.
• Probability ≠ clinical risk. ECE 0.049 is measured, not assumed but a softmax score is still not a calibrated disease probability for one patient.
• Resources: trained on a free Kaggle GPU, served on CPU with a 28 MB ONNX file no fine-tuned foundation model.