Did Microsoft’s AI Beat Doctors? The Paper Measured Something Specific

🔍 Read the full analysis: Did Microsoft’s AI Beat Doctors? The Paper Measured Something Specific on ThorstenMeyerAI.com

Before you orderOffer from Amazon

Get hardware and tech essentials delivered free with Prime

• Fast, free delivery on millions of items
• Prime Video, Amazon Music and more included
• Member-only deals all year

Start your free Prime trial →Free trial for eligible customers · Cancel anytime

As an affiliate, we earn on qualifying purchases.Hide

Microsoft’s MAI-DxO diagnostic AI scored 83.2% on 304 complex medical case records in a study published October 22 in Science, compared with 36.1% for generalist physicians who could use textbooks and the internet. The peer-reviewed result concerns historical, text-only cases in a simulator—not diagnoses or outcomes for real patients—so it does not show that the system outperforms doctors in clinical practice.

The paper, “Sequential diagnosis with language models,” evaluated more than whether a system could name a diagnosis from a complete case file. In the Sequential Diagnosis Benchmark, each case began with a short summary. The doctor or AI had to request additional information, such as answers to questions or test results, from a simulated gatekeeper holding the rest of the record. The researchers assessed both the final diagnosis and the estimated cost of obtaining information.

Microsoft reported that MAI-DxO scored 83.2% on the 304 complex cases, while physicians scored 36.1% with reference materials and 22.3% without them. On a separate set of 325 emergency department cases, the reported scores were 80.3% for the system and 46.8% for physicians. These are results reported by Microsoft’s research team, whose 15 authors were affiliated with the company; publication in Science means the paper underwent peer review, not that independent researchers have reproduced its findings.

MAI-DxO is an orchestration system that uses a language model as a panel of virtual physicians. The agents propose possible diagnoses, request the next question or test, and challenge one another over multiple rounds. Microsoft’s chart also reports that GPT-5 alone scored 78.9% on the complex cases, against 83.2% for MAI-DxO. That comparison suggests most of the measured accuracy came from the underlying model, while the system added about four percentage points, according to the source material. It also reports lower estimated test costs for the orchestrated approach, but the supplied account does not give a complete cost comparison across all study groups.

Accuracy Is Not Clinical Performance

The study addresses a real challenge in medical AI: diagnosis often involves deciding what information to seek next, not simply selecting an answer after every fact has been provided. A benchmark that charges for requested tests can measure both diagnostic performance and the resources used to reach a conclusion. That makes the work more informative than a simple multiple-choice test, while still leaving a substantial gap between a simulation and patient care.

For hospitals and clinicians, the reported scores are a reason to examine the method and await clinical testing, not evidence that the system is ready to replace medical judgment. The comparison was between a multi-agent AI setup and individual generalists working alone, rather than specialists or physician teams. The cases were selected as difficult teaching cases, and the NEJM set contained no healthy patients or harmless explanations. The study therefore could not measure how often the system would raise false alarms in a broader patient population.

For patients, the distinction matters because diagnostic accuracy on a curated record does not establish whether using the tool improves treatment, reduces harm, shortens time to diagnosis, or works fairly across different groups. Those outcomes require testing in clinical settings. The paper is a benchmark result—not evidence of improved patient outcomes.

From Case Benchmark to Patient Study

The work appeared in Science, volume 394, pages 423–428, on October 22, 2026. An earlier, non-peer-reviewed version was posted on arXiv in June 2025. That preprint used an earlier system based on OpenAI’s o3 and reported a larger gap: 80% for the AI against 20% for doctors. The published paper’s figures and setup should be distinguished from those earlier results.

Some methodological details described in the source account come from the preprint, because the full final paper is behind a paywall; those details may differ from the Science version. In that earlier account, a model judged diagnoses against the published answer, and physicians’ re-grading showed good but imperfect agreement. When an ordered test had no recorded result, the gatekeeper generated a synthetic result. Test prices were estimated using billing codes and a 2023 U.S. hospital price list, with a fixed $300 physician visit cost. The authors characterized these costs as first-order approximations.

The physician comparison also requires care. The preprint described 21 U.S. and UK generalists—17 in primary care and four hospital generalists—with a median of 12 years’ experience. They were assessed on the 56 most recent held-out cases, while the AI was assessed on all 304. The published figures include physician averages across case attempts, but differences in how the groups were tested limit direct interpretation of the headline comparison.

““I really believe that in 2027, we will be on the verge of a coding style breakthrough for medical records.””

— Microsoft AI chief executive Mustafa Suleyman, speaking to TechCrunch

Limits of the Doctor Comparison

It remains unclear how well MAI-DxO would perform with real patients, complete clinical workflows, and routine cases. The supplied material does not report a completed prospective trial or independent replication. It also does not establish whether the AI’s higher benchmark score would translate into better patient outcomes, fewer missed diagnoses, or acceptable rates of unnecessary tests and false alarms.

The comparison’s groups were not equivalent in every respect: the AI system used multiple agents, while the physicians worked individually, and specialists or clinical teams were not included. The benchmark’s model-based grading, synthetic results for unavailable tests, and estimated prices also affect how the scores and cost claims should be read. Details taken from the earlier preprint may not match the final paper in every respect.

Prospective Testing Alongside Care

The next reported step is a prospective study at Beth Israel Deaconess Medical Center in Boston. According to the source material, consenting patients will interact with the system alongside their usual care. That setting can test how the tool behaves with real clinical information and workflows, rather than historical case records alone.

Results from that study will be needed to assess safety, practical usefulness, and effects on care. Until they are available, the published accuracy figures should be understood as benchmark results from simulated cases. The source material does not specify when the prospective study will begin, how many patients it will enroll, or when findings are expected.

Key Questions

Did Microsoft’s AI beat doctors in real-world care?

The study did not test real-world care. It compared MAI-DxO with generalist physicians on historical medical cases in a text-based simulator. Whether the system improves diagnosis or outcomes for actual patients remains unestablished.

What did the 83.2% score measure?

It was Microsoft’s reported diagnostic accuracy for MAI-DxO on 304 complex case records. The benchmark required the system to request additional case details and tests before making a diagnosis; it was not a measure of patient survival, treatment success, or general clinical performance.

How did physicians perform in the study?

Microsoft reported physician scores of 36.1% with textbooks and internet access and 22.3% without those references on the complex cases. The comparison was with individual generalists, not specialist teams, and should be read alongside differences in the testing setup.

Has the result been independently confirmed?

The paper was peer-reviewed and published in Science, but the supplied material does not report an independent reproduction of the findings. Peer review is not the same as confirmation by another research group.

What test is planned next?

A prospective study at Beth Israel Deaconess Medical Center is reported as the next step, with consenting patients interacting with the system alongside normal care. Its start date, enrollment size, and expected reporting date have not been specified in the source material.

Source: ThorstenMeyerAI.com