The Highland Marketing advisory board met to consider the government’s enthusiasm for AI. To date, healthcare has mostly experimented with decision support tools, and their impact on the NHS and UK plc has been mixed. But the big, new idea is generative AI; and members felt some careful thought is needed about how and where to adopt it.
Ask Google’s Gemini assistant how many articles have been published on AI in the UK over the past year, and the response is that the data is “too vast and dispersed” to say, but “it’s a very active area of discussion.”
Or, as Jeremy Nettle, chair of the Highland Marketing advisory board, said it’s almost impossible to avoid news about AI and new uses of AI just now: and not all of it is good.
“There has been some pretty horrific AI coming out of the US about Trump, and Gaza,” he said, referring to a widely circulated video showing a golden statue of the US president in a Las Vegas-style Gaza strip.
Putting the full weight of the British state behind AI
This has not stopped the government making a big bet on AI. Last July, UK science secretary Peter Kyle asked tech entrepreneur Matt Clifford to conduct an AI ‘opportunities review’ and in January the government accepted its recommendations.
These range from creating the power and water infrastructure needed for new data centres, to setting up a library of public data sets to train new models, to creating ‘growth zones’ and running ‘scan, pilot, scale’ projects’ in public services.
Prime minister Sir Keir Starmer made a speech promising to put “the full weight of the British state” behind the plans. And a week later, the government published a ‘blueprint for modern, digital government’ with a big role for AI – and a suite of tools for civil servants, called Humphrey.
But haven’t we been here before?
While Sir Keir’s speech made a splash, consultant and former NHS CIO Neil Perry pointed out that we’ve heard similar things before. “Previous governments have talked about being leaders in technology and investing in new UK companies, and we’re not seeing it come through,” he said.
As a case in point, the advisory board considered the development of AI in the NHS, which has so far mainly focused on clinical decision support tools and automated reporting for x-rays and scans.
Entrepreneur Ravi Kumar, whose company CyberLiver works on digital therapies for advanced liver disease, said this requires a huge amount of development work, regulatory compliance, and liaison with clinicians.
“Getting AI into a practical, implementable state to work in a clinical setting is very expensive,” he said. “So, when the government talks about creating a home-grown industry, we have to be realistic about the number of companies that will be able to persist long enough to make the grade.”
NHSX, the digital unit set up by former health secretary Matt Hancock, recognised this problem. It set up a National Artificial Intelligence Lab to “bring together academics, specialists and health tech companies” to “work on the biggest challenges” facing the system.
While it did some interesting work, advisory board members felt its record was spotty when it came to growing UK businesses. Some prominent health-AI companies have lost out to over-seas competitors or left for bigger markets themselves. And there’s no money on the table for anything similar, today.
Decision support tools: company and clinician experience has been mixed
From a clinical perspective, radiology expert Rizwan Malik also felt mistakes had been made. “Millions have gone into AI in the NHS, particularly in the diagnostic imaging space,” he said. “But we gave a lot of companies money to develop things without asking what the business case was.
It was: ‘AI is the answer, now what’s the question?’” As an example, he said AI is good at detecting abnormalities in chest x-rays, but so are radiologists, who now have to check both the chest x-ray and the AI’s interpretation of it. Which adds to their workload, for unclear benefit.
“That’s why I say we need business cases,” he said, underlining that those business cases need to think about how AI can help individuals to work smarter, not just harder.
“At the moment, it’s always: ‘how many more scans will you be able to do with this, Rizwan?’ It’s never: ‘how will this make your practice safer’? Or: ‘how will it improve outcomes?’ Or: if it can do those things, is it worth the price tag?”
Coming soon: generative AI
The AI that everybody is talking about right now, though, is generative AI or the large language models like OpenAI’s ChatGPT, Google’s Gemini, and the open-source Llama.
These take large-scale data inputs and use them to predict what character is most likely to follow another character, creating (or generating) new outputs in the process, such as a response to a web query, or a block of code, or a social media post.
Jason Broch, a GP and CCIO, said he was worried about putting LLMs into clinical spaces. “We have used Microsoft Copilot for administrative work,” he said. “Some of the people who take minutes at meetings have tried it, and they have found it can cut the time involved in producing a report from three hours to one hour.
“But that is because they are experts at producing reports. In a clinical setting, we don’t know whether the output from an LLM is good, or not.”
People are using AI assistants because they are free or, increasingly, built into consumer software packages. But they’re not transparent. “We don’t really know what data [a model] has been trained on,” he said. “It produces an output, but we don’t really know how it does that.
“If you run a prompt again, it can come up with a completely different output. We need guardrails for the use of LLMs. Or we need healthcare specific models, because if we are going to scale the use of these tools in the NHS, we need to be able to trust them.”
Sam Neville, a nurse and CNIO, agreed. “Trust is an important word,” she said. “Staff do not trust this technology, and patients don’t trust it either.
“If we tell patients that we are going to put their information into a third-party system like a patient portal, they don’t like it. If we tell them that the NHS is looking at AI, they think Trump video. They think we are just going to make things up.”
Where’s the regulation, where are the guardrails?
David Hancock, a consultant and interoperability expert, said he is worried that the government is paying far too little attention to these issues, in its dash for growth and productivity gains.
The EU, he pointed out, has passed legislation (the EU AI Act) to ban certain uses of AI and encourage transparency and labelling. Whereas the UK’s approach does not have the same level of emphasis on human rights protections.
“The UK government has said that it sees not being in the EU as an opportunity, so it sounds as if it is not going to go down the same route,” he said. “It looks as if it will allow this to be more commercially driven, as it is in the US.”
Nicola Haywood-Cleverly, a former CIO and consultant who also works as an NHS non-executive director, felt the NHS also needs to think much harder about the data that is being fed into these tools. “We all know there is a lot of concerns regarding data quality out there,” she said. “If we want to train good models, we need better data to train them on.”
The NHS will also need better infrastructure, she added, to make sure new tools are properly embedded into clinical workflows, and clinicians are clear about when they are using AI outputs.
Neil Perry said this raised the question of how the NHS can make sure new tools are implemented safely. “I have just joined one of NHS England’s panels looking at refreshing DCB 0129 and 0160 [clinical risk management standards for companies and organisations looking to roll-out digital systems].
“One of the first questions asked was: is the standard fit for AI? And the answer is: not really. In fact, it’s not really fit for two-week sprints [software development cycles]. When DCB 0129 and 0160 were written, the NHS was lucky if it got a system update yearly. We need to refresh methodologies. And we need to educate and include clinicians and patients.”
Jane Brightman, a social care expert who works at Skills for Care, said social care staff and people drawing on care also need to be brought into the picture. The social care sector is doing some work with the University of Oxford on the “ethical use of AI” that should lead to some basic principles for its development and deployment.
Time to think clearly
Jason Broch also suggested that the NHS needs to avoid some of the mishaps that it has made with AI to date by thinking clearly about what LLMs are good at and where that can resolve some of the challenges that the NHS is facing.
“We need to get cleverer about language,” he said. “We talk about LLMs as if they generate meaning. But they don’t. We talk about ‘hallucinations.’ But the LLM isn’t hallucinating. It’s doing what it’s meant to do, it’s just that we don’t like the output. So, we need to understand that these things are a great language tool, but they are not a cognition tool.”
Following on from this, he suggested the best uses of generative AI in the NHS might be in helping with language tasks, such as summarising a mass of patient records before an appointment, or generating communications.
Advisory board members had many other ideas for using AI alongside other technologies. Sam Neville said she is looking at an AI tool that can review trends in outpatient appointments to identify patients who may be at risk of ‘DNAing’ or not attending appointments.
David Hancock said the NHS could usefully run something similar over its patient reported outcomes or PROMS data, to find out what it is getting for its money.
Highland Marketing chief executive Mark Venables said it is working with an AI firm that can take vital signs information from patients waiting for admission and alert clinical teams to signs of deterioration.
Neil Perry suggested that similar technology could be used in A&E, to make long waits safer. “We can argue about whether all of this is AI, or whether it is just technology,” he said. “The point is that it automates what we do anyway, accurately enough to trigger an alert that leads to a human decision.”
Build out, take people with you
The biggest problem, he said, is that in the current NHS financial environment projects like this are difficult to implement. He argued that instead of making big statements about AI, the government should focus on where it could address the big “volume” issues and use its buying power to secure solutions for the whole system.
“Back in the days of the national programme for IT we used to talk about ‘ruthless standardisation’,” he said. “Perhaps we could do a bit of that now. Build AI tools into the NHS App and 111 services to detect and diagnose conditions, or read vital signs from a selfie and direct patients to the most appropriate service.
“The technology is available; we need to make it meaningful, useful and used at scale.” Meanwhile, Rizwan Malik argued there were some good things to have come out of the faltering start that has been made on AI so far.
“The upside is that we have experience of decision support tools,” he said. “So, perhaps we can start talking about the best way to use them. Instead of sending everybody for an MRI or CT scan we can start talking about which patients really need them. Or which patients need to go first.
“We could make incremental improvements. For the millions invested so far, we cannot say we are at the forefront of AI in healthcare in the UK, or that we are supporting UK plc. But we do have a workforce ready to have meaningful conversations going forward.”
