AI Model Transparency

How to read the development metrics

Each number above was measured on a held-out validation split of that model's own development dataset. mAP@50 is mean average precision at 50% overlap between predicted and labeled regions. Precision is the share of predicted regions that matched a labeled tumor. Recall is the share of labeled tumors the model found. These are not real-world accuracy measurements. The three models do different tasks on different datasets, so their numbers can't be compared with each other.

Why real-world results are much lower

• Familiar images: The validation split comes from the same curated collection as the training data. It measures familiar-looking images, not scans from new hospitals.
• Scanners: Scanner vendor, magnetic field strength and reconstruction settings change brightness, contrast, noise and resolution. The same anatomy can look different from one machine to another.
• Sequences and protocols: Original Detection was trained on DWI, but uploads can be T1, T2, FLAIR or contrast-enhanced images, with different slice thickness, orientation and slice position.
• The image path: Uploads may be screenshots, phone photos or crops with text overlays, different windowing and JPEG compression that the training images did not have.
• Patients and case mix: Patients differ in age, may have post-surgical changes or conditions the models never saw. The classification models only know glioma, meningioma and pituitary tumors, so they must choose one even when none applies.
• One slice vs. a full study: The models see one 2D slice. Radiologists read full 3D studies with several sequences, earlier scans and clinical history.
• Confidence is not probability: Model confidence scores are not calibrated probabilities. A model can be confidently wrong on images unlike its training data.
• Evaluation design: A general lesson in medical imaging research is that random image-level splits can put near-identical slices from the same patient in training and validation, making scores look better than they are. Patient-level splits and external test sets from other hospitals give more honest estimates; this is not a claim that such overlap happened here.

What we've learned from real uploads

Reviewing scans uploaded to this site, especially ones flagged with "Report Incorrect", shows that the models frequently fail on real-world scans: they can miss a tumor, flag normal tissue or predict the wrong tumor type. Strong development metrics did not transfer to real uploads. Data diversity and evaluation design matter as much as model architecture. This project is learning how to make models more adaptable across hospitals, scanners and patient populations. No real-world accuracy has been measured yet.

What we're working on (planned goals)

Today, only the "Report Incorrect" button and admin review of those reports exist. The following are goals, not features available today:

• More diverse training data from multiple sites and MRI sequences.
• Augmentation that simulates scanner differences and image-quality problems such as noise, blur, compression and contrast changes.
• Consistent preprocessing and intensity normalization before images reach the model.
• Patient-level data splits and testing on external data the models have never trained on.
• Confidence calibration and detecting uploads unlike the training data.
• Using reviewed "Report Incorrect" cases to find recurring failure patterns.