-
Technology
-

AI is Taking Over NHS Hospitals and the Hidden Safety Flaws Will Shock You

By
Distilled Post Editorial Tem

Over the coming months, a consultant at Barnsley Hospital will be able to finish an outpatient appointment without having typed a word. A microphone will have captured the conversation, software will have drafted the clinical notes and GP letter, and the consultant will simply review, amend and sign the documents. For parts of the NHS, this is already becoming familiar practice. Barnsley Hospital NHS Foundation Trust has now moved beyond its initial trials, awarding Lyrebird Health a £38,400 contract for ambient voice technology licences running from 2 November 2026 to 2 February 2028. The pilots in emergency care and outpatients have paved the way for a wider commitment. Yet the more pressing question is what happens when the software gets something wrong.

Barnsley's own board papers recognise that artificial intelligence cannot simply be introduced and left to run. Its approach treats governance as an ongoing responsibility, which is both sensible and resource-intensive. Clinical safety assessments, data protection reviews and regular checks on the accuracy of AI-generated records all require specialist attention. That work falls to clinical safety officers, information governance leads and digital teams already occupied with electronic patient record programmes and other commitments. Every trust procuring the technology must find the capacity to carry out much of this work itself.

Barnsley is far from alone. Four trusts in south west London have contracted with the same supplier, Lyrebird Health, in a rollout intended to reach up to 20,000 clinicians over four years. Leicester and Northamptonshire have taken a different route, procuring Accurx Scribe, powered by Tandem Health, for more than 10,000 clinicians. Mid Cheshire has extended its use of ambient voice technology across additional hospital locations and community services following a successful trial. Meanwhile, the North West London Acute Provider Group, which covers four trusts, has begun market engagement for a proposed three-year contract running from 1 April 2027 to 31 March 2030. National planning guidance is urging providers to deploy the technology at pace, while the neighbourhood health framework extends the ambition into general practice. The appetite for adoption is unmistakable.

The supply market, however, is proving harder to navigate. NHS England's supplier registry, launched in January, now lists 30 companies that have self-certified against its requirements. It gives hospitals a starting point when considering potential suppliers, but inclusion should not be mistaken for a guarantee of safety or formal NHS England endorsement. The responsibility for local assurance and procurement decisions remains with individual organisations. That distinction matters. Earlier this year, the Health Service Journal reported that an NHS trust had abandoned plans to trial a major American supplier's product after concerns were raised about its compliance with NHS England's assurance requirements. The episode illustrates how much depends on individual trusts having the expertise to scrutinise what is being offered. Meanwhile, the Health Services Safety Investigations Body has opened an investigation into ambient voice technology in hospitals, and the medicines regulator has issued guidance explaining how medical device rules apply. Both developments are timely, particularly for trusts that have already committed to contracts.

Wales offers a revealing comparison. Digital Health and Care Wales had planned an open framework agreement with an estimated value of £10 million, but abandoned the procurement on 20 October after too few suppliers met its Conditions of Participation. The decision can be read as a refusal to compromise on standards. It also exposes the difficulty of setting requirements that a developing market can realistically satisfy. A pre-qualified supplier list is of little use if only a handful of companies can make it through the process. Wales has a single national digital body, an arrangement often held up as a model for English systems, yet its attempt to establish a central procurement framework still stalled. The lesson is uncomfortable but clear: requirements must be rigorous enough to protect patients while remaining achievable for suppliers operating in the market today. Without that balance, trusts risk returning to individual procurement, with little of the protection central arrangements were supposed to provide.

For trust boards, the financial and operational implications deserve closer attention. Business cases for ambient voice technology tend to focus on the time returned to clinicians and the speed with which letters reach patients and GPs. Those benefits are real, but the calculations can overlook the cost of assurance. Sampling AI-generated records for errors, maintaining audit trails and monitoring performance over time all require staff and funding. A board approving an 18-month or three-year licence should be asking who will take responsibility once the original pilot team has dispersed and the early enthusiasm has faded. There is another question, too: what happens when a supplier updates its software or underlying model? A product that passed a safety assessment in the spring may not behave in quite the same way by winter.

National bodies face a different challenge. Having dozens of NHS organisations repeat near-identical safety assessments for similar products is an inefficient use of expertise that is already in short supply. NHS England has begun developing national support through its guidance and wider ambient voice technology programme, but local responsibilities remain substantial. A shared, regularly refreshed assessment of each registered product, which trusts could use and supplement with their own local checks, would turn much of this duplicated work into a common resource. Suppliers would also have a clearer set of expectations to meet. The abandoned Welsh framework shows why getting those requirements right matters: standards that are either too loose or too difficult to satisfy will do little to help the market mature.

Technology companies should expect a more demanding NHS customer as experience grows. Trusts that have moved beyond their first deployment will want evidence that suppliers can detect changes in performance, explain how software updates are communicated and provide reliable audit trails. They will also expect contracts to make responsibilities clear when an AI-generated record contributes to patient harm. Companies whose business models assume that hospitals will shoulder the entire safety burden may find that assumption increasingly difficult to defend. The ability to produce a clinical note quickly will no longer be enough to secure confidence.

Patients, meanwhile, remain at the end of a process whose risks they may barely see. Clinicians are expected to review and verify AI-generated documentation before it becomes part of the patient record, but legal responsibility is more complicated than placing it entirely on the professional who signs the letter. Trusts retain important duties of care, while suppliers may also carry responsibilities depending on the circumstances and their contractual obligations. The practical difficulty remains the same: the technology is being introduced to relieve clinicians of administrative pressure, yet those clinicians must still find the time to scrutinise its output. If that review becomes little more than a glance, the promised efficiency could come at the expense of patient safety.

The time saved by ambient voice technology is a compelling reason for the NHS to explore its potential. Doctors should be able to spend less of their working day typing and more of it listening to patients. But the long-term value of that change will depend on the systems surrounding the software. National guidance and emerging support arrangements are beginning to take shape, yet trusts still carry much of the responsibility for deciding whether a product is safe and monitoring how it performs. Until national assurance can offer more consistent and dependable support, every new contract like Barnsley's will remain a local decision with consequences extending well beyond the hospital that signs it.