Replication, Replication, Replication

25 June 2009

Epidemiological studies may be subject to various types of bias, including publication bias (where studies with statistically significant results are more likely to be published than those without) and time-to-publication bias (where studies with significant results are more likely to be published more quickly than those without). These two forms of bias are well-documented and result in an abundance of spuriously positive study results in the published literature. Ioannidis and Trikalinos [Ioannidis and Trikalinos. (2005) J Clin Epidemiol 58: 543-549] have previously shown that an additional bias, the so-called ‘Proteus phenomenon’, often occurs very early on during the accumulation of scientific evidence, which results in extreme and sometimes contradictory findings. This phenomenon highlights the importance of separate groups performing validation studies in independent data sets, a practice now acknowledged as a prerequisite for convincing epidemiological evidence. As the evidence accumulates and evolves, it should be collated and summarised systematically, with careful control over biases and chance effects, in order to identify true associations and interactions amongst the large pool of false positives (see previous news).

Nowhere is this problem more apparent than within genetic epidemiology. Reported genetic associations with common complex disease have become numerous in the published medical literature, and because the risks are generally of small magnitude, replication is absolutely essential for weeding out false positive findings and avoiding bias. However, the paucity of established gene-environment interactions to date reflects a widespread failure to incorporate both the genetic and environmental factors in a joint analysis, which weakens the observed association between a true risk factor and the disease and may help to explain the small magnitude of associations often observed thus far.

A recent systematic review has now disproven one of the few long standing gene-environment interactions [Risch et al. (2009) JAMA 301(23): 2462-2471]. In 2003, an association was established for a “depression gene”, serotonin transporter gene 5-HTTLPR, and depression, showing that in combination with stressful life events, genetic variation in the promoter region of the 5-HTTLPR gene plays a role in predisposition to major depression [Caspi et al. (2003) Science 301(5631): 386-389]. Caspi et al. found that individuals with one or two copies of the short allele appeared to be more prone to depression following a stressful life event (such as sudden unemployment). This finding was particularly attractive because it offered a plausible biological link. Six years on, following several subsequent studies by various study groups, Risch et al. have systematically reviewed the evidence on this topic in order to summarise how the evidence has evolved. Risch et al. identified 26 potentially eligible studies of which 14 met the criteria for a meta-analysis. In their analysis, Risch et al. found no evidence to support the claim that the 5-HTTLPR genotype either alone or in interaction with stressful life events is associated with an elevated risk of depression. Their analysis shows that the stressful life events themselves have a significant association with the risk of depression.

Comment: This systematic review and meta-analysis highlights two important issues. First, the importance of replication studies conducted using independent data sets by independent groups. Second, as studies are published, the accumulating evidence should be systematically appraised allowing true associations to be identified amongst the ever increasing pool of published associations.

More from us

Genomics and policy news