Electronic Letters to:

Review Article: A. Petrie Statistics in orthopaedic papers J Bone Joint Surg Br 2006; 88-B: 1121-1136 [Abstract] [Full text] [PDF]

eLetters: Submit a response to this article

Electronic letters published:

Author's reply

Aviva Petrie   (27 October 2006)

Statistical stumbling blocks

Richard H Browne   (27 October 2006)

A meta-analysis is not a systematic review

Yu-Min Lin, Tai-Sheng Tan, Tu-Sheng Lee   (22 September 2006)

Author's reply 27 October 2006
Aviva Petrie, Statistician UCL Eastman Dental Institute Send letter to journal: Re: Author's reply a.petrie{at}eastman.ucl.ac.uk Aviva Petrie	Sir, I would agree with Dr Tan et al that the terms meta-analysis and systematic review should not be used interchangeably. However, I take exception to the remark in their e-letter which suggests that I have done so in my review article, and apologise if there has been a misunderstanding. The review article states that a meta-analysis is "a systematic review of the literature that seeks to identify, appraise, select and synthesise all high-quality research relevant to the question of interest and uses quantitative methods to summarise the results". The first part of the sentence describes the systematic review in general and the last phrase, "and uses quantitative methods to summarise the results", describes the particular aspect of a meta-analysis that defines its specific property that qualifies it as a distinct type of systematic review. This accords with Dr Tan’s definition of a meta-analysis, as well of those of the Toronto and Oxford Centres for Evidence-Based Medicine,1,2 which define it, respectively, as a "systematic review that uses quantitative methods to synthesize and summarize the results" and "a systematic review or overview which uses quantitative methods to summarize the results". Furthermore, by incorporating quantitative methods into the systematic review, I maintain that the meta-analysis should retain its position at the pinnacle of the hierarchy of evidence. The hierarchy moves from simple observational procedures at the bottom to increasingly sophisticated and statistically refined methods at the top. The meta-analysis has the facility, unlike a qualitative systematic review, to investigate statistical variation (heterogeneity) in the results from the different studies and, if there is no evidence of worrying heterogeneity, to use the variability associated with the estimates obtained from different studies to provide an overall estimate of the effect of interest, with an associated confidence interval. However, it should be noted that the structure of the hierarchy of evidence diagram in the review article is not definitive for two reasons. Firstly, it only provides a brief outline of the hierarchy and, as a consequence, not all types of study are included in it. For example, it omits the general systematic review of randomised controlled trials (RCT), the systematic review of case-control studies, non-blinded RCTs and cohort studies with a poor follow-up. Secondly, the relative positions of the components of the hierarchy depend to some degree on the problem at hand, as indicated in the review article, and the extent to which a correct and proper methodology has been applied to the process of investigation. Thus, a meta-analysis with statistical heterogeneity should not be regarded as superior to a qualitative systematic review in terms of the evidence provided, nor should some systematic reviews be viewed as providing better evidence than a large and well-conducted RCT. A. Petrie, Statistician, Biostatistics Unit, UCL Eastman Dental Institute, London, UK. 1.Toronto Centre for Evidence-Based Medicine.http://www.cebm.utoronto.ca/ (accessed 27/10/06) 2.Oxford Centre for Evidence-Based Medicine. http://www.cebm.net/ (accessed 27/10/06)
Statistical stumbling blocks 27 October 2006
Richard H Browne, Biostatistician Texas Scottish Rite Hospital for Children Send letter to journal: Re: Statistical stumbling blocks richb{at}tsrh.org Richard H Browne	Sir, I commend Dr Petrie on her comprehensive article on biostatistics in orthopaedics. I would like to offer some addenda based on my experience. 1. Significance testing can only tell us whether we have successfully ascertained the direction of the difference of two means or two proportions. This is seen by examining the associated 95% confidence interval for the difference of the two means or proportions. If p<0.05, then all of the differences will have the same sign. Conversely, if p>0.05, the confidence interval will have some differences positive and some negative. So, p<0.05 should be associated with a statement such as: 'We are 95% confident that the mean effect of Treatment A is greater than the mean effect of Treatment B.' For p>0.05, we should say: 'We are unable to say whether Treatment A is superior to Treatment B, or vice versa,' or 'The results are inconclusive with respect to which treatment is the best.' Note that we never claim that the treatment effects are equal when p>0.05, but only that the direction has not yet been determined. Also, we have made no claim as to the magnitude or clinical importance of the difference. 2. Any claim of clinical importance of a difference should be accompanied by the lower 95% confidence limit of the difference. Even if the difference of the sample means or proportions is clinically important, that does not mean that the true difference is also clinically important. It is quite possible that a difference in proportions of 0.30 is highly significant, but that the lower bound of the confidence interval is only 0.04, a difference of no clinical importance. Without the lower bound, we can only say that a large difference has unconfirmed clinical importance. R.H. Browne, Biostatistician, Texas Scottish Rite Hospital for Children, Dallas, Texas, USA.
A meta-analysis is not a systematic review 22 September 2006
Yu-Min Lin, Orthopaedic Surgeon Department of Orthopaedics, Taichung Veterans General Hospital, Tai-Sheng Tan, Tu-Sheng Lee Send letter to journal: Re: A meta-analysis is not a systematic review ymlin{at}vghtc.gov.tw Yu-Min Lin, et al.	Sir, The article by Petrie is a good primer for the biostatistics. However, we have some concern about the hierarchy of evidence as shown in the figure 1. In the text, the author used the terms 'meta-analysis' and 'systematic review' interchangeably. This usage will confuse novices. Literally speaking, systematic review (also known as systematic overview) is defined as a review of a clearly formulated question that uses systematic and explicit methods to identify, select, and critically appraise relevant research, and to collect and analyse data from the studies that are included in the review.  Statistical methods (meta-analysis) may or may not be used to analyse and summarise the results of the included studies, whereas meta-analysis is defined as the use of statistical techniques in a systematic review to integrate the results of included studies. It is sometimes misused as a synonym for systematic reviews, where the review includes a meta-analysis.1 The top of the hierarchy of evidence should be systematic reviews of randomised controlled trials,2 not meta-analysis. Tai-Sheng Tan, MD, Tu-Sheng Lee, MD, PhD, Yu-Min Lin, MD, MMS, Department of Orthopaedics, Taichung Veterans General Hospital, Taichung, Taiwan. 1. Green S, Higgins J, ed. Glossary. Cochrane Handbook for Systematic Reviews of Interventions 4.2.5 [updated May 2005]. http://www.cochrane.dk/cochrane/handbook/handbook.htm [accessed 7/9/06]. 2. Greenhalgh T. How to read a paper: the basics of evidence-based medicine. Third ed. Malden: BMJ Books/Blackwell Publishing Ltd, 2006:15-39.