³Ô¹ÏÍø

Artificial intelligence is transforming healthcare, but new research involving ³Ô¹ÏÍø shows it may also put some patients at far greater risk of having their data exposed.

Artificial intelligence is increasingly used to support diagnosis and treatment in healthcare. However, a new study published in Nature, involving researchers from Imperial College London’s Department of Computing, shows that these systems can unintentionally reveal whether a person’s data was used to train them.

Working with collaborators at the Technical University of Munich and the Hasso Plattner Institute, researchers from ³Ô¹ÏÍø’s Department of Computing, including Professors and and ³Ô¹ÏÍø alumnus Moritz Knolle, investigated how these risks arise in practice. In one of the first patient level privacy audits of medical AI, they examined how well membership inference attacks can identify whether a specific individual’s data was included in a model’s training set.

At the Department of Computing, we are committed to ensuring that as these technologies become more powerful, they deliver benefits fairly and safely for all people. It also underscores the value of international collaboration, such as our strategic partnership with the Technical University of Munich." Professor Stefanos Zafeiriou Professor in Machine Learning and Computer Vision and Head of the Department of Computing at ³Ô¹ÏÍø

Here, an “attack” does not mean hacking. Instead, it refers to testing whether an AI model can reveal if someone’s data was used to train it. This might seem like a small detail, but in healthcare the implications can be significant, as Professor Ben Glocker explains: “If a model has been trained on a broad population, identifying whether someone’s data is included may reveal little. But for models built using more specific groups, such as patients with a particular disease or from a single hospital, that same information can act as a direct signal of sensitive medical data. For example, a successful attack against a model which predicts anti-cancer immunotherapy efficacy from routine blood test data, could reveal that an individual has cancer.”

While previous research has measured these risks on average across datasets, the team found that this can mask much higher risks for certain individuals. Across a range of medical datasets, the researchers showed that such attacks can achieve near perfect success rates for some patients, even when overall results suggest systems are broadly safe.

The study also found that these risks are not evenly distributed. Patients from underrepresented groups, including those defined by disease, demographic characteristics or how their data was collected, were significantly more likely to be identified. At the same time, the number of patients at high risk increased as AI models became larger and more complex.

Together, the findings show that commonly used approaches to measuring privacy can underestimate individual risk. The researchers highlight the need for stronger safeguards, including techniques such as differential privacy, which introduce carefully designed noise during training so that models learn general patterns without exposing information about any one individual. By ensuring that no single patient’s data stands out, these approaches could help reduce the risk of identification, even for those with rare or distinctive characteristics.

“This work, led by PhD student Moritz Knolle, reflects a central focus of our research: the safe deployment of medical imaging AI in real-world settings. It shows that standard privacy evaluations can overlook serious risks to individual patients, particularly those from underrepresented groups. and that these risks may grow as models become more powerful. By introducing patient-level auditing and demonstrating practical safeguards, we provide a clear path toward medical AI systems that are not only accurate, but also trustworthy, fair, and safe to deploy at scale.”

, Head of ³Ô¹ÏÍø’s Department of Computing said: “This Nature paper highlights the importance and the responsibility of applying advanced AI to real-world healthcare challenges. At the Department of Computing, we are committed to ensuring that as these technologies become more powerful, they deliver benefits fairly and safely for all people. It also underscores the value of international collaboration, such as our strategic partnership with the Technical University of Munich. Studies like this are essential to making sure progress in AI is matched by trust, equity, and public good.”

Read the full