This article is part of our series, The Clinician’s Perspective, where we explore the intersection of AI and healthcare through the eyes of our team – former clinicians who understand the realities of patient care firsthand.

The safety standard already exists. Most AI scribes haven't met it

Many customers raise general concerns with us about safety, asking how they could be sure the tool wasn't hallucinating or quietly inventing things in a patient note. It is exactly the right question to ask! However, what customers often don’t know is that the concern already has an answer: there are standards built precisely for this, and there are products on the market that meet them. In our case, Tandem is regulated as a medical device under the European Medical Device Regulation (EU MDR), which means notified bodies have assessed, documented, and externally validated the safety questions customers are worried about before deployment.

Think about it this way: when you buy a bike helmet, you don't necessarily need to check that it's been tested to absorb impact and actually protect your head. You assume the market only offers ones that are, because someone decided helmets are worth regulating and unsafe ones don't get sold. AI medical scribes haven't reached that point yet. A clinician can't yet assume that every product on the market has cleared a safety bar, because many haven't, and some vendors are actively arguing they shouldn't have to. So the burden still falls on the buyer to ask the right question and look for the right things.

The regulation hasn't changed. The technology has.

AI didn't arrive in healthcare with a new rulebook. The European Medical Device Regulation (EU MDR) was already there. EU MDR regulates devices based on their intended use and not based on the technology used. So concretely, an AI scribe that informs patient management is a medical device (not because it uses AI, but because of what it does to a patient's care pathway).

What changed in the Era of AI Software is how easy it became to build - among other things - AI medical device prototypes that perform well enough to appear safe and effective. With no safety evaluation. No clinical standards. Often nobody even asked whether those things were needed.

Under EU MDR, any AI tool that summarises clinical information based off a patient encounter and thus informs patient management is a medical device. It needs to be conformity-assessed, fulfil general safety and performance requirements, CE marked, and maintained with a technical file, a clinical evaluation, and an ongoing post-market surveillance programme. This is indeed a continuous obligation for the lifetime of the product.

Most AI scribes on the market today have not done this work.

Classification matters more than most people realise

Under EU MDR, medical devices are classified by risk. Tandem's AI Medical Scribe, Coding Assistant, and Clinical Decision Support tool are all classified at least at Class IIa in the EU due to their intended purpose of informing patient management. That requires independent conformity assessment by a notified body before a product can be placed on the market. In fact, software rarely falls below Class IIa. MDCG 2019-11 rev.1 makes that explicit. It's a bar most AI scribes haven't cleared. Most haven't even tried.

In more detail, during the conformity assessment a notified body examines the specific use case, real-world safety and performance, and every process the manufacturer runs to maintain quality: risk management, continuous improvement, product monitoring. It covers the product and the organisation behind it.

On the topic of classification, Sweden's Medical Products Agency (Läkemedelsverket) made the regulatory picture explicit earlier this year, when it initiated a formal market surveillance programme targeting AI scribes specifically. The findings confirmed what the regulation already said: software that informs patient management is a medical device. It must be classified correctly and go through the proper approval channels.

A Class IIa certification from a notified body means safety and performance have been examined by someone independent. There's no credible substitute for this assessment. While some vendors argue compliance slows innovation, I'd argue that compliance ensures the innovation is safe and can have long term impact. The speed of getting a product to market doesn't justify skipping the process that exists to protect patients and ultimately also the manufacturer.

Safe and useful are two different problems

In my experience around regulatory compliance, there are two compliance approaches manufacturers take. One of these is purely defensive, where passing audit and getting certification is the goal, but that is not what building a clinical product requires.

The primary consideration should be safety and performance which happens to also be compliant. In fact, safety and performance are two distinct obligations under EU MDR itself.

In the AI era, the bar for what good performance looks like has certainly moved. AI enables faster iteration, which means poor usability is harder to excuse. Healthcare customers should be demanding good UX, not treating cognitive load and clunky workflows as an acceptable trade-off. A product that passes the audit but makes documentation slower, or produces notes a clinician has to rewrite, has certainly failed the performance bar.

Another aspect of AI products is that they behave differently from traditional software. They're non-deterministic. They can evolve. As the model updates, as usage shifts, as the clinical context changes, the safety and performance profile can change too. Proper post-market surveillance for AI software is the only mechanism that keeps the product safe over time. At Tandem, every piece of feedback that touches safety or performance gets analysed, escalated, and fed into improvements alongside our product monitoring and QA setup that ensures clinical safety and continuous improvement. In contrast, most vendors treat it as a compliance box to tick, which makes it both an untapped resource for improvements and also indicates that manufacturers are not taking sufficient responsibility for their products.

Compliance isn't a team. It's how you build.

You cannot build a compliant medical device if compliance is treated as someone else's problem.

In most healthcare technology companies I have encountered, compliance runs behind product development. Or it gates development. It gets called in when procurement asks for documentation, or when a regulator arrives. Either way it's a blocker to move past, not a foundation to build on.

At Tandem it works the other way. The real work of quality management isn't producing compliant documentation. It's making sure the whole organisation understands the why behind the rules and how they fit the way the company actually operates. When that's true, the documentation follows naturally and one truly has fostered a quality culture.

What to ask any AI vendor

If you're a clinical leader evaluating AI, asking "are you CE marked?" is a good start.

Additionally, I would ask whether the product is correctly classified under EU MDR, and whether the manufacturer can show the conformity assessment documentation (e.g. Declaration of Conformity, QMS certification). Ask how they handle post-market surveillance. Find out what happens if the product contributes to a clinical incident. Ask whether they hold ISO 13485, the standard that governs quality for software as medical devices.

The bar is going up

Regulation is tightening. The EU AI Act is going to introduce additional obligations for high-risk AI in healthcare that sit alongside EU MDR requirements. The interaction between the two frameworks is still being worked through, but the direction is clear.

The practical reality right now: a clinician using a non-IIa AI product may be using something that was built in days, never independently tested for safety or accuracy, and never seen by a third party. It's sadly the current state of most of the market.

In contrast, companies that have done the foundational work are well placed for the future. The ones that haven't will find catching up is harder than starting right.

To conclude: Clinical AI works. The potential to reduce documentation burden, support accurate coding, improve how we interact with patients, and give clinicians more time to treat and diagnose is already being realised. But it works when compliance is built into the product from day one, so please ask for it!

About Dr Yan Peng Zhao

Dr Yan Peng Zhao is Head of Medical Compliance at Tandem Health and a former trauma surgeon. The AI Medical Scribe, Coding Assistant, and Clinical Decision Support tool are all certified at EU MDR Class IIa. Tandem is certified under ISO 13485, ISO 27001, and ISO 42001, and CE marked under EU MDR.