Microsoft's 83% diagnostic AI score compares the wrong doctors

Microsoft's AI diagnostic system scored 83% accuracy on some of medicine's hardest cases. The doctors it was tested against scored 36%. That headline number is real, peer-reviewed, and published in Science. It's also built on a comparison that no hospital would ever run in practice — and if you're evaluating clinical AI for your health system or insurtech platform, the gap between the number and the reality is exactly where the risk lives.

On Thursday, Microsoft published a study in Science showing that MAI-DxO, its multi-agent Diagnostic Orchestrator, correctly diagnosed complex medical cases 83% of the time in simulation. The comparison group: primary care doctors and general internists working individually, with access to reference materials. TechCrunch reported that the cases came from the New England Journal of Medicine — the kind of diagnostic puzzles that stump even experienced clinicians. Without reference materials, generalist physicians scored just 22%, according to The Verge.

MAI-DxO works by coordinating several AI agents that behave like a panel of doctors: questioning a patient, ordering tests, debating a diagnosis. Microsoft tested it with models from OpenAI, Google, Anthropic, Meta, and DeepSeek. Its best results came with OpenAI's GPT-5 and GPT-5.5. With GPT-5.5, DxO pushed accuracy on complex cases from 72% to nearly 87%. On emergency department cases drawn from Beth Israel Deaconess Medical Center's patient database, DxO scored 80% versus an average of 47% for human physicians — though on those same ED cases, GPT-5.5 alone scored about 85% with or without DxO, which raises its own questions about what the orchestration layer is actually adding.

Microsoft AI CEO Mustafa Suleyman called the result 'a major milestone' toward what the company's manifesto defines as 'medical superintelligence' — 'the capacity to surpass the strongest contemporary human-led health systems in medical reasoning and coordination across the full arc of care.' Microsoft plans to test DxO in real hospital settings through a new partnership with Beth Israel Deaconess Medical Center, with Dominic King, Microsoft AI's VP of Health, saying the study will likely 'get underway in the next couple of months,' though neither he nor Suleyman committed to a firm date.

The comparison that changes everything

Here's the problem. The cases used in the study are, as TechCrunch noted, of a complexity that is usually handled by specialists or physician teams. Those specialists were not part of the comparison. So Microsoft ran a multi-agent AI system — coordinated, iterative, with unlimited compute and no time pressure — against solo generalists tackling cases that would, in any real hospital, be escalated to a specialist team long before a final diagnosis was made.

That's not a small methodological footnote. It's the whole ballgame. Asking a solo primary care doctor to diagnose a rare autoimmune presentation from a NEJM case report is a bit like testing a Formula 1 car against a commuter sedan and declaring the future of transportation solved. The sedan driver wasn't the right comparison to begin with.

To be fair, the 80%-vs-47% result on actual Beth Israel ED cases is harder to dismiss. Real patient admissions, not curated puzzles. That comparison still pits DxO against human physicians generally rather than the specialist teams who'd handle the toughest cases, but it's closer to a real-world scenario. And the researchers note that 'improvements in model capability do not necessarily diminish the benefits' of multi-agent coordination — meaning the system isn't just riding GPT-5's coattails.

The cost numbers also deserve a look before anyone gets too excited. GPT-5 alone with a basic prompt scored 78.9% on complex cases — at about $8,400 per case. DxO cut that cost by about $3,000 and raised accuracy by only about four points. That's a better ratio, but $5,400 per diagnostic case is not a number that fits inside any realistic clinical workflow today. The peer-reviewed paper also doesn't include results for the newest frontier models — GPT-6 Astra, Anthropic's Opus 5.5, Google's Gemini 4 Argon, Meta's Muse Wonder Full — so the competitive picture is already dated.

What this means if you're making decisions now

Suleyman's framing — 'it is finally here,' 'next year we're going to see extremely impressive deployments' — is the kind of language that gets clipped into board presentations and vendor pitches. Don't let it short-circuit your evaluation process.

If you run or advise a health system, an insurer, or any business where clinical AI is on the roadmap, three things are worth doing this week. First, ask any vendor citing this study which physician cohort their benchmark used. 'Outperforms doctors' means something very different when the doctors are solo generalists versus specialist teams. Second, check whether the system has been tested prospectively on live patients or only on historical cases — the Beth Israel real-world study hasn't started yet. Third, get clarity on compute costs. A system that costs thousands of dollars per case isn't a clinical tool; it's a research prototype.

The skeptic's best argument is that even a flawed comparison can point at something real. If MAI-DxO genuinely helps a generalist physician in a rural clinic catch a rare diagnosis they'd otherwise miss, the comparison group matters less than the outcome. That's a fair point. But 'this might help in underserved settings' and 'this outperforms doctors' are very different claims, and only one of them is being made in the press release.

Watch for the Beth Israel prospective study results. That's the number that will actually tell us something. If DxO's recommendations align with specialist teams at a meaningful rate — not just beat solo generalists — then Suleyman's milestone claim holds up. If the gap closes when real specialist panels are the comparison, the 83% figure will look a lot more like marketing than medicine. My bet is the real-world results will be genuinely useful and substantially more modest than this week's headlines suggest. The question is whether hospital procurement teams will wait to find out.