Black box medicine and transparency

20 November 2018

 

 

 

 

 

Artificial intelligence is increasingly used throughout health systems, supporting healthcare professionals to make diagnoses through imaging and pathology and to optimise decisions about treatments and patient management.

Together with some complex algorithms, some of these uses are examples of ‘black box medicine’ in which opaque computational models are used to make decisions related to healthcare. These models can help speed up diagnoses as well as develop new treatments but, due to the amount of data they use and their complexity, their ‘reasoning’ cannot be explicitly understood or even stated.

The PHG Foundation has been awarded seed funding from the Wellcome Trust to examine what ethical and legal rules should, and could, apply to black box medicine. Clarifying the requirements for transparency and explanation could help to improve patient and public trust in how these technologies are used in healthcare.

This project consists of three phases that explore different aspects of transparency and explanation relating to the use of algorithms in healthcare:

Phase 1: We will undertake a philosophical evaluation of the ethical principles that apply to transparency in black box medicine

Phase 2: We will analyse what is legally required. In particular, we will explore the scope, extent, and impact of the general principle of transparency and right to explanation under the General Data Protection Regulation, and, where applicable, the UK Data Protection Act 2018. We will then assess competing explainable machine learning approaches to determine whether they meet these requirements

Phase 3: Using our knowledge from phases 1 and 2 of what is ethically and legally required, we aim to develop a tool to assist developers to think through the practical, ethical, and legal reasons to make their machine learning model human interpretable.

Outputs from this project will include reports, briefing materials and an academic paper.

Timeline

  • November 2018

    Project initiation

  • June 2019

    Roundtable on black box medicine and transparency: developers

  • June 2019

    Report published: A right to explanation?

  • July 2019

    Roundtable on black box medicine and transparency: clinical focus

  • July 2019

    Report published: Why explainable machine learning matters for health

  • September 2019

    Expert Roundtable on Black Box Medicine and Transparency: Policy and regulatory focus

  • November 2019

    Report published: Interpretability by design

  • February 2020

    Final report published: Black box medicine and transparency

 

For more details of this project please contact Alison Hall or Johan Ordish

See also Regulating algorithms in healthcare

Genomics and policy news