PRISM2 reads 2.3M pathology slides and talks back, beating clinical-grade cancer detectors

Here’s a stat that should make any pathologist sit up: 2.3 million whole-slide images. That’s the training diet for PRISM2, a new Microsoft and Paige collaboration that doesn’t just classify cancer — it answers diagnostic questions about it. The model was trained jointly on tissue tiles and 685,507 real pathology reports from Memorial Sloan Kettering, which GPT-4o converted into Q&A pairs. The result isn’t a pixel classifier. It’s a system that aggregates thousands of tile embeddings per slide into a single representation and then generates text.

The architecture makes a specific bet. Stage one trains a perceiver-based slide encoder with two loss functions running simultaneously: a contrastive objective using BioGPT text embeddings, and an autoregressive objective driven by Phi-3 Mini. The idea is that contrastive training alone makes embeddings good for retrieval but lousy for generation, while autoregressive training alone can memorize text patterns without learning useful visual features. Doing both against the same encoder output is meant to split the difference — and it seems to work.

PRISM2 exposes two separate embeddings, and which one you use depends on the job. Base embeddings come straight from the slide encoder and transfer well to biomarker prediction. Diagnostic embeddings get pulled from the hidden state of the 4-billion-parameter language model after it processes slide latents alone, with no text prompt involved. On pan-cancer detection, diagnostic embeddings hit a 0.967 AUC against 0.956 for the base embedding. PRISM scored 0.947, TITAN came in at 0.931. The gap tightens on rare cancers, where sparse training examples drag the diagnostic embedding down to 0.957.

What’s genuinely surprising is how this holds up under linear probing — the cleanest read on representation quality because it holds the encoder fixed and tests whether a simple classifier can extract signal. PRISM2 embeddings never statistically underperform prior foundation models across the diagnostic benchmarks tested, and they improve most where training data is scarcest. Survival prediction tells a similar story: fine-tuned PRISM2 embeddings beat a specialist model trained from scratch on over 225,000 cases, widening the gap to a 0.809 concordance index on colorectal cancer recurrence-free survival. Even the base embeddings hold their own on survival tasks without any task-specific tuning. The single-turn dialogue supervision, however, means this system can’t yet manage the back-and-forth a deployed clinical consult would actually demand.