Black box medicine and transparency

20 November 2018

 

 

 

 

 

Artificial intelligence is increasingly used throughout health systems, supporting healthcare professionals to make diagnoses through imaging and pathology and to optimise decisions about treatments and patient management.

Together with some complex algorithms, some of these uses are examples of ‘black box medicine’ in which opaque computational models are used to make decisions related to healthcare. These models can help speed up diagnoses as well as develop new treatments but, due to the amount of data they use and their complexity, their ‘reasoning’ cannot be explicitly understood or even stated.

The PHG Foundation has been awarded seed funding from the Wellcome Trust to examine what ethical and legal rules should, and could, apply to black box medicine. Clarifying the requirements for transparency and explanation could help to improve patient and public trust in how these technologies are used in healthcare.

This project consists of three phases that explore different aspects of transparency and explanation relating to the use of algorithms in healthcare:

Phase 1: We will undertake a philosophical evaluation of the ethical rules that apply to transparency in black box medicine

Phase 2: We will analyse the requirements of laws and regulations to clarify what is legally required. In particular, we will explore the scope, extent and impact of the right to explanation in the General Data Protection Regulation and similar provisions in the UK Data Protection Act 2018. We will then assess competing theoretical approaches, such as counterfactual explanation to determine whether they meet these requirements.

Phase 3: Using our knowledge from phases 1 and 2 of what is ethically and legally required, we aim to develop exemplars of model explanations that can be implemented across a variety of black box medicine applications such as imaging (radiology or histopathology) and decision support systems used by healthcare professionals. We will test these in three separate meetings of doctors, patients, developers and policy makers.

Outputs from this project will include reports, briefing materials and an academic paper.

Timeline

  • November 2019

    Project completion

  • November 2018

    Project initiation

For more details of this project please contact Alison Hall or Johan Ordish

See also Regulating algorithms in healthcare