Writing in the Lancet, Ioannidis et al have analysed the results of 55 meta-analyses of genetic association studies, to see if there was evidence that the magnitude of the genetic effect found in different studies depends on the size of the study or varies markedly between the first-published study and later ones [Ioannidis, JPA et al (2003) Lancet 361, 567-571]. They report that for somewhere between 20% and 40% of the meta-analyses (depending on the statistical method used to assess the heterogeneity) there was indeed a significant difference between the size of effect measured in large and small studies – in most cases, the larger studies found a more modest genetic effect. Similarly, in about a quarter of the meta-analyses the first-published study reported a stronger effect than was found in subsequent ones. Overall, only 16% of the associations that had been subjected to meta-analysis appeared to be robust and without evidence of heterogeneity or bias. The authors comment that the failure of reported associations to stand up to meta-analysis may be due to real genetic heterogeneity (for example, the association may vary in different populations) or to one or more of many possible sources of bias in individual studies.
Comment: This paper adds further strength to calls for all reported genetic associations to be viewed cautiously until they are confirmed by multiple replications and/or by well-designed studies involving large numbers (preferably thousands) of both cases and controls. (This theme is also explored in the following Journal Club item.)