Reliability of risk of bias assessment of RCTs in systematic reviews and meta-analyses
Systematic reviews and meta-analyses of randomised controlled trials (RCTs) are the gold standard in clinical evidence and are central to evidence-based healthcare decision making. In particular, meta-analysis is increasingly becoming a vital component of health technology assessment processes. However, a meta-analysis is only as robust as the RCTs that inform the dataset; inclusion of RCTs that are subject to high levels of bias may result in a biased meta-analysis. Therefore, the quality of individual trials should be taken into account when deciding which RCTs to include in the meta-analysis, and trials deemed to have a high risk of bias should be considered for exclusion.
The Cochrane Collaboration tool (1) is a widely employed method used to assess the risk of bias in RCTs. The tool is comprised of six domains (including random sequence generation, allocation concealment, blinding, and selective outcome reporting), which are categorised as low, high, or unclear risk of bias for each RCT. Often, incomplete reporting of study methodology in the trial publications introduces challenges in determining the overall risk of bias for a study. However, a good understanding of how to correctly implement and interpret the tool is required to ensure reliable and accurate results. A recently published study investigated the effectiveness of different levels of training on the reliability of Cochrane risk of bias assessments by inexperienced analysts (2). In the study, two pairs of analysts were randomised to either intensive standardised training or to minimal training. The agreement between each group of analysts and reference scores from a pair of experienced analysts, and the agreement between individuals in each pair of analysts, were then assessed.
The results indicated that the reliability of the Cochrane Collaboration risk of bias tool assessments may be improved if analysts receive standardised training, including instructions and decision rules on how to correctly classify trials for each domain. The authors recommended that training should also be tailored for specific research areas to address the common methodological concerns associated with particular topics (2).
This research highlights the importance of a good understanding of RCT methods, the challenges associated with the specific therapeutic research area of interest, and how to correctly implement the appropriate quality assessment tools when appraising trials for the risk of bias.
- Higgins JPT, Altman DG (editors). Chapter 8: Assessing risk of bias in included studies. Cochrane Handbook for Systematic Reviews of Interventions. Chichester (UK): John Wiley & Sons. 2008.
- da Costa BR, Beckett B, Diaz A, Resta NM, Johnston BC, Egger B, Jüni P, Armijo-Olivo S. Effect of standardized training on the reliability of the Cochrane risk of bias assessment tool: A prospective study. Systematic Reviews. 2017;6:44.