·

·

12

min read

ChatGPT in clinical decision support: capabilities and limitations

Explore ChatGPT's role in clinical decision support: where it helps, where it fails, and how to evaluate AI tools safely in healthcare

Consultant psychiatrist conducting a mental state examination, listening to a patient and making notes on a structured assessment form in an NHS consulting room

ChatGPT can support clinical workflows, but it wasn't designed, trained, or validated as a medical device. Understanding where it genuinely helps, where it falls short, and what governance needs to surround its use is now a practical necessity for anyone working in healthcare.

What clinical decision support actually means in 2026

Clinical decision support (CDS) is a spectrum, not a single technology. At one end sit passive, rule-based alerts: a prescribing system flagging a contraindicated drug combination, or a reminder that a patient is overdue for a cervical smear. At the other end are active, AI-driven tools that interpret unstructured clinical data, generate differential diagnoses, or recommend treatment pathways in real time.

That distinction matters when it comes to assessing risk. A rule-based alert that fires incorrectly causes alert fatigue. A large language model (LLM) that confidently produces a plausible but incorrect differential diagnosis in a complex case can contribute to patient harm. ChatGPT and its equivalents occupy a distinctive position on this spectrum: they're extraordinarily capable at language tasks, but they weren't designed or validated as medical devices.

How large language models work: what clinicians need to know

LLMs like ChatGPT generate text by predicting the most statistically probable next word or phrase, based on patterns learned from vast quantities of training data. They don't reason the way a clinician reasons. They don't access a structured knowledge base, consult a pharmacopoeia, or apply logical rules. They generate responses that are linguistically coherent and often factually accurate, because accurate information was well-represented in their training data.

This is fundamentally different from rule-based CDS systems. A rule-based system will always flag a specific drug interaction if the logic is correctly coded. An LLM may or may not flag it, depending on how the question is phrased, what was in its training data, and the inherent variability of its probabilistic outputs. LLMs are sophisticated pattern-matching engines that produce language, not lookup systems with a reliable index, and clinicians should evaluate their outputs accordingly.

Where ChatGPT shows genuine promise

A scoping review of 28 studies published in Health Science Reports identified clinical decision support and medical education as the most commonly cited advantages of ChatGPT in healthcare. The evidence is strongest in a few specific areas.

Clinical documentation is the clearest win, and the gap between documentation and everything else is wider than the literature sometimes implies. A synthesis published in Frontiers in Artificial Intelligence found 40–70 per cent time savings in documentation tasks when LLMs assisted with note generation and summarisation. LLMs also show measurable utility in drafting patient letters, explaining diagnoses in plain language, and speeding up literature summarisation, though all outputs require verification.

The case for supporting non-specialists is more nuanced but meaningful. A June 2026 study in NPJ Digital Medicine evaluated ChatGPT-4o on 100 real-world polyneuropathy cases, finding 65.5 per cent accuracy for the leading diagnosis, comparable to non-specialist neurologists at 63.0 per cent, though lower than specialists at 74.0 per cent. Non-specialists revised their assessments in 21.8 per cent of cases after reviewing ChatGPT-4o's suggestions, and their accuracy improved as a result. The model also outperformed non-specialists on differential diagnoses (82.0 per cent versus 77.5 per cent) and recommended more appropriate confirmatory tests (68.0 per cent versus 53.0 per cent). A systematic review of LLMs across the colorectal cancer care continuum, published in the Journal of Medical Internet Research in May 2026, found similar utility in automating data extraction from clinical texts and supporting clinical decision-making, with domain-specific and multimodal models showing advantages over general-purpose models in certain tasks.

The accuracy problem: hallucinations, calibration failures, and known weak spots

The most clinically significant limitation of LLMs is hallucination: generating plausible-sounding but factually incorrect information. A November 2025 analysis of hallucination risks in LLMs deployed in healthcare found that hallucinated outputs can appear credible and steer clinicians towards harmful interventions. Even minor inaccuracies can escalate clinical risk when they align with an existing clinical suspicion, because the model confirms what the clinician already suspects and the error goes unchallenged.

The hallucination problem is compounded by a confidence calibration failure. A comprehensive evaluation of four LLMs on complex rheumatology cases, published in Health Informatics Journal in April 2026, found uniformly weak, non-significant associations between reasoning quality and expressed certainty across all models. The models didn't reliably know when they were wrong, and expressed similar confidence regardless of whether their reasoning was sound. ChatGPT had the lowest hallucination rate of the four models tested, at 7.4 per cent, compared to Gemini's 18.5 per cent. That relative advantage is real, but 7.4 per cent represents a meaningful error rate in a clinical context.

Three additional failure modes deserve specific attention. Knowledge cutoffs mean that clinical guidelines, drug approvals, and safety alerts issued after the training data cutoff won't appear in responses, unless the model has access to live retrieval tools. Performance degrades in complex cases: a simulation-based study published in Healthcare (MDPI) in July 2025 found that ChatGPT's reliability declined significantly in cardiovascular cases with comorbidities, precisely the patient population most likely to be seen in secondary care. Drug interaction detection is also unreliable: a European study published in Clinical Pharmacology & Therapeutics in February 2025 found that ChatGPT produced different answers in 90 per cent of cases when the same drug–drug interaction query was repeated and missed clinically relevant interactions, a finding strong enough that the study authors concluded it can't currently be recommended for this use case at all.

Patient safety risks clinicians should not overlook

Automation bias, the tendency to defer to an AI-generated suggestion even when clinical judgement would have led to a different conclusion, is well-documented in the human factors literature and applies directly to LLM use. The calibration failures described above make this worse: a model that expresses uniform confidence gives clinicians no signal about when to push back.

Performance in sequential decision-making is substantially worse than in static question-answering. The AgentClinic benchmark, published in NPJ Digital Medicine in April 2026, found that LLM diagnostic accuracy in sequential, interactive clinical scenarios can fall to below one-tenth of the accuracy achieved on equivalent static questions. This matters because real clinical encounters are sequential and dynamic, while most benchmark evaluations are not. A model that looks capable on a written case vignette may behave very differently when it has to gather information, revise its assessment, and act across multiple steps.

Rare diagnoses present a particular vulnerability. LLMs are trained on population-level data, and rare conditions are by definition underrepresented in that data. A study evaluating five LLMs on 50 multiple myeloma clinical scenarios, assessed by three independent haematologist-oncologists, concluded that LLM treatment recommendations require careful clinical supervision to ensure patient safety, even in a structured, oncology-specific evaluation context.

Regulatory status and data governance: the accountability gap

General-purpose LLMs like ChatGPT aren't marketed with a medical intended purpose, so they don't carry CE marking as medical devices under the EU Medical Device Regulation (MDR). When a clinician uses consumer ChatGPT to support a clinical decision, there's no validated evidence base, no manufacturer liability framework, and no regulatory audit trail. Clinical accountability rests entirely with the clinician.

The data governance risk is equally underestimated, and in day-to-day practice it may be the more immediate concern. Under the UK General Data Protection Regulation (GDPR) and the EU's GDPR, patient data is special category data requiring an explicit legal basis for processing. Entering a patient's name, date of birth, clinical history, or test results into a consumer LLM tool almost certainly means a third-party controller is processing personal data without a data processing agreement, without a lawful basis, and potentially with data held outside the UK or EU. Enterprise agreements with OpenAI or equivalent providers can include data processing agreements, commitments not to use inputs for model training, and defined data residency. Consumer-facing tools carry none of these protections by default, raising significant data security and privacy concerns.

Before using any AI tool clinically, verify four things: whether a data processing agreement is in place between your organisation and the AI provider; where patient data is processed and stored; whether your organisation's information governance team has approved the tool; and whether patient data can be de-identified before input without losing clinical utility.

How purpose-built clinical AI differs from general-purpose LLMs

The difference between purpose-built clinical AI and a general-purpose LLM lies in the entire deployment architecture, not simply the underlying model. A benchmark study published in NPJ Digital Medicine in December 2025 tested six LLMs across 30 metrics covering CDS domains. Domain-specific medical LLMs showed consistent performance advantages over general-purpose models, with higher top scores in both safety (0.912 versus lower general-purpose scores) and effectiveness (0.861). Overall performance across all models averaged just 57.2 per cent, with a 13.3 per cent performance drop in high-risk clinical scenarios.

Purpose-built clinical AI tools designed for regulated healthcare environments typically offer validated outputs tested against clinical outcomes, audit trails that support clinical governance and accountability, medical record system integration, and medical device certification under relevant regulatory frameworks, specifically MDR in the EU and UKCA marking or registration with the Medicines and Healthcare products Regulatory Agency (MHRA) in the UK. The underlying language model may be similar or identical to a general-purpose LLM. What differs is the governance layer, the validation evidence, and the regulatory accountability.

What the evidence actually says: important caveats

The published evidence on LLM performance in clinical tasks is growing rapidly but remains methodologically uneven. LLMs have achieved scores in the 80–90 per cent range on medical licensing examinations such as the United States Medical Licensing Examination (USMLE), a result that attracted significant media attention and tells us considerably less than it appears to. Licensing examinations test recall and structured reasoning on well-defined problems; they don't replicate the ambiguity, time pressure, incomplete information, and sequential decision-making of real clinical encounters.

Study quality is variable. The systematic review of LLMs in colorectal cancer care found that only 27 per cent of included studies had low risk of bias, with problems in outcome measurement, patient selection, and lack of blinded assessment. Evaluation frameworks aren't yet standardised: a 2025 expert consensus published in ScienceDirect identified the absence of consistent evaluation methods as a significant barrier to safe deployment. Performance also varies substantially by specialty. The evidence base is strongest in areas with large, well-structured training datasets, including common presentations in primary care, oncology staging, and radiology interpretation, while performance in rare diseases, complex multi-morbidity, and time-sensitive acute presentations remains less well characterised.

Appropriate use versus inappropriate reliance: where to draw the line

The evidence supports a clear distinction. Low-risk, high-volume tasks such as documentation assistance, patient letter drafting, and literature summarisation are where LLMs offer the most benefit with the least risk, provided patient data governance requirements are met. High-stakes diagnostic or prescribing decisions, particularly in complex, multi-morbid, or rare-disease presentations, require validated, regulated tools with defined intended uses and clinical oversight built into the workflow. Consumer tools without enterprise data agreements should never be used with identifiable patient data, regardless of the clinical task.

Across the literature, human expert oversight is treated as essential, but it's worth being direct about what that means in practice. Given current LLM accuracy, calibration failures, and the absence of regulatory validation for general-purpose tools, oversight is a structural requirement with no foreseeable expiry date, not a temporary safeguard pending better AI. The Annals of Biomedical Engineering study on ChatGPT in CDS reached this conclusion, as did the ChatGPT-CARE preprint, which identified specific limitations including the tendency to ignore clinical practice guidelines, lack of reasoning transparency, and overly general responses. In-context learning and chain-of-thought prompting, techniques that guide the model through explicit reasoning steps, are proposed as mitigation strategies, but these remain directions of travel for purpose-built tools rather than solutions available in consumer deployments.

When evaluating any clinical decision support tool, the key questions are: whether it's been validated on comparable patient populations; what the published evidence says on hallucination rate and performance in high-risk scenarios; whether it carries medical device certification; and whether your organisation's information governance and clinical safety teams have approved it. A commentary published in the Journal of the American Medical Informatics Association (JAMIA) in January 2025 acknowledged both the promise and the significant gaps in evidence, noting that key limitations and areas for future research remain unresolved. The tools are improving and the regulatory frameworks are evolving, but until validated clinical AI demonstrates consistent, safe performance across the full range of clinical complexity, the clinician remains the essential checkpoint in any AI-assisted decision.

Frequently asked questions

▶ Is ChatGPT a regulated medical device for clinical decision support?

No. Because ChatGPT is not marketed with a medical intended purpose, it falls outside the EU Medical Device Regulation entirely. That gap has a practical consequence: if something goes wrong, there is no manufacturer liability framework to invoke and no regulatory audit trail. The clinician carries the full accountability.

▶ What clinical tasks show the strongest evidence for LLM utility?

Documentation is the clearest win by some distance, with 40 to 70 per cent time savings reported in the Frontiers in Artificial Intelligence synthesis. After that, the evidence thins quickly. Gains in differential diagnosis support and literature summarisation are real but smaller, and they depend heavily on the clinician actively reviewing and correcting outputs rather than accepting them.

▶ What is hallucination, and why does it matter in clinical settings?

Hallucination means a model produces confident, fluent text that is factually wrong. The specific danger in clinical use is that an incorrect output confirming an existing suspicion is the one least likely to be challenged. ChatGPT's 7.4 per cent hallucination rate was the lowest across four models tested in the Health Informatics Journal evaluation, but that still means roughly 1 in 14 outputs contains an error.

▶ Can ChatGPT reliably detect drug–drug interactions?

No. The Clinical Pharmacology and Therapeutics study found different answers in 90 per cent of repeated identical queries, which means the output a clinician receives depends partly on how the question is phrased that day. For drug interaction checking, use a rule-based prescribing system with consistently coded logic instead.

▶ What are the GDPR risks of entering patient data into consumer LLM tools?

Before entering any patient data into a tool, confirm three things: whether your organisation has a signed data processing agreement with the provider, where the data is stored and processed, and whether your information governance team has approved the tool. Without all three, you are almost certainly processing special category data without a lawful basis.

▶ How does a purpose-built clinical AI tool differ from general-purpose ChatGPT?

The underlying model may be similar. What differs is everything around it: validation evidence tested against clinical outcomes, audit trails for governance, medical record system integration, and medical device certification. The NPJ Digital Medicine benchmark found domain-specific medical models scored 0.912 on safety metrics against lower general-purpose scores, with overall performance across all models averaging just 57.2 per cent.

▶ Does passing medical licensing exams mean ChatGPT is safe for clinical decision support?

Licensing exam scores measure recall on well-defined problems, which is a narrow slice of clinical practice. The AgentClinic benchmark found diagnostic accuracy in sequential, interactive scenarios can fall to below one-tenth of accuracy on equivalent static questions. Real clinical encounters are sequential and dynamic, so exam performance is a poor proxy for bedside reliability.

▶ What is automation bias, and how does it affect clinicians using AI tools?

Automation bias is the tendency to defer to an AI suggestion even when clinical judgement points elsewhere. It is made worse by the calibration failure documented in the Health Informatics Journal evaluation: because models express similar confidence whether their reasoning is sound or not, clinicians receive no signal about when to push back.

▶ Where should clinicians draw the line between appropriate use and inappropriate reliance on LLMs?

A practical test is whether an error in the output would reach a patient before a clinician could catch it. Documentation drafts, patient letters, and literature summaries allow review before any action is taken. Diagnostic or prescribing decisions in complex cases do not offer that buffer, and consumer tools without enterprise data agreements add a data governance breach on top of the clinical risk.

Get started with Tandem today

Join thousands of clinicians enjoying stress-free documentation.

Get started with Tandem today

Join thousands of clinicians enjoying stress-free documentation.

Get started with Tandem today

Join thousands of clinicians enjoying stress-free documentation.