20 November 2018
Artificial intelligence is increasingly used throughout health systems, supporting healthcare professionals to make diagnoses through imaging and pathology and to optimise decisions about treatments and patient management.
Together with some complex algorithms, some of these uses are examples of ‘black box medicine’ in which opaque computational models are used to make decisions related to healthcare. These models can help speed up diagnoses as well as develop new treatments but, due to the amount of data they use and their complexity, their ‘reasoning’ cannot be explicitly understood or even stated.
The PHG Foundation has been awarded seed funding from the Wellcome Trust to examine what ethical and legal rules should, and could, apply to black box medicine. Clarifying the requirements for transparency and explanation could help to improve patient and public trust in how these technologies are used in healthcare.
This project consists of three phases that explore different aspects of transparency and explanation relating to the use of algorithms in healthcare:
Phase 1: We will undertake a philosophical evaluation of the ethical principles that apply to transparency in black box medicine
Phase 2: We will analyse what is legally required. In particular, we will explore the scope, extent, and impact of the general principle of transparency and right to explanation under the General Data Protection Regulation, and, where applicable, the UK Data Protection Act 2018. We will then assess competing explainable machine learning approaches to determine whether they meet these requirements
Phase 3: Using our knowledge from phases 1 and 2 of what is ethically and legally required, we aim to develop a tool to assist developers to think through the practical, ethical, and legal reasons to make their machine learning model human interpretable.
Outputs from this project will include reports, briefing materials and an academic paper.
Roundtable on black box medicine and transparency: developers
Report published: A right to explanation?
Roundtable on black box medicine and transparency: clinical focus
Report published: Why explainable machine learning matters for health
Expert Roundtable on Black Box Medicine and Transparency: Policy and regulatory focus
Report published: Interpretability by design
Final report published: Black box medicine and transparency
See also Regulating algorithms in healthcare