Why explainable machine learning matters for health

Machine learning is a form of AI that combines massive datasets and advanced processing power with statistical models to develop a system that 'learns' how to perform a specific task, without explicit instructions. In healthcare, machine learning has many potential applications, though its use raises a number of concerns.

Key amongst the many potential applications of machine learning is diagnostics and treatment. But many commentators fear that machine learning threatens to usher in an age of black box medicine, where computer systems make decisions and diagnose patients in an opaque manner, without proper explanation.

This PHG Foundation discussion paper explores the use of machine learning for health, why it can be considered to be 'opaque' and why this matters, and outlines the different ways in which machine learning models can be made interpretable to humans.

Key points

• Machine learning models vary in extent of interpretability - from those that are intrinsically human interpretable to closed black boxes which are not
• Black box models can be somewhat interpretable through the use of post hoc (after the event) explainers of particular decisions or explanation of general functions
• Post hoc explainers have weaknesses, as they are ultimately only an estimation of the underlying model. This makes it unclear where and when we should demand the use of intrinsically interpretable machine learning

This work was supported by the Wellcome Trust, grant number: 213623/Z/18/Z

By Johan Ordish, Alison Hall