Statistics from Altmetric.com
Traumatic injury to the CNS is a leading cause of death and acquired disability worldwide. It is estimated that well over 10 million people experience head injuries each year,1 of whom several thousand will die, and several thousand will be permanently disabled. The identification of effective strategies in the prevention, treatment, and rehabilitation of injury of the CNS is of considerable public health importance. Because the validity of effectiveness information from randomised controlled trials is far more readily assured than with other study designs, the randomised controlled trial has become the gold standard for evaluating treatment effectiveness. Many healthcare interventions have only moderate effects on outcome. However, for a problem as common as traumatic brain injury, even moderate effects can have important public health implications. Moderate treatment effects, whether favourable or adverse, are difficult to assess reliably, and require large amounts of randomised evidence. This evidence can be obtained in two ways: by conducting large scale prospective randomised trials, and by systematic review and meta-analysis of the evidence from previous trials.
Finding all relevant randomised controlled trials is undoubtedly the major challenge of systematic reviews. Inclusion of all relevant randomised trials increases the power of a systematic review to detect moderate treatment effects, and also avoids the mistaken inference that can arise when a biased selection of trials is reviewed. For example, in defence of the assertion that steroids are ineffective in the treatment of severe brain injury, a recent report cited the results of three randomised trials.2 These three trials comprised only a small proportion (three out of at least 17 trials) of all randomised trials on this topic. Although a systematic review may have reached the same conclusion, it is not acceptable to leave ourselves open to the bias of selecting trials just because we like the results.3
Searching out the randomised evidence
Identifying all the randomised trials on any particular topic is far from straightforward. Although a proper Medline search is essential, even the best searching will miss a substantial proportion of trials. It has been shown that when the gold standard is all known randomised trials published in journals indexed in Medline and those not indexed in Medline, the sensitivity of Medline is only 51%.4 This is mainly because many trials are not properly indexed. As a result, systematic reviews that use Medline searches alone, such as those published by the Brain Trauma Foundation, are likely to miss at least half of all trials. Because of the poor sensitivity of bibliographic medical databases, it is necessary to supplement database searches with hand searching of journals and conference proceedings. Hand searching involves the page by page searching of the entire text of a journal issue, including articles, abstracts, news columns, editorials, and letters, to find all randomised controlled trials. Hand searching can take a considerable amount of time and effort, but is often the only way to completely identify published reports of trials. For example, in three neurosurgical journals (indexed in Medline), hand searching identified 156 reports of randomised controlled trials. Of the 156 reports, only 42% had a record in Medline, and only 26% were identifiable as trials using the Publication Type headings “randomized-controlled-trial” or “controlled-clinical-trial.” Within the Cochrane Collaboration, the details of all trials that are identified by hand searching are extracted and incorporated into a central registry of trials, so that once a journal has been hand searched it need never be searched again.
Despite the use of hand searching, many randomised trials never contribute to the scientific basis of health care, for the simple reason that they have never been formally published. Unpublished data may include trials that have never been submitted for publication; or unpublished clinically relevant outcomes, from an otherwise published trial. Not only does the failure to include unpublished trials in a systematic review result in statistical imprecision in estimating treatment effects, but because trials that show more promising treatment effects are more likely to be submitted for publication, reviews based on published studies alone may give misleading conclusions about treatment effectiveness.5 6 The extent of the problem of unpublished trial data is illustrated by a systematic review of steroids in acute traumatic brain injury. In this systematic review, more than a fifth of all the available randomised evidence was unpublished, comprising data on some 445 randomised patients in the third and fourth largest trials ever done on this topic.7
Some systematic reviews exclude trials published in languages other than English. Once again this represents a unnecessary waste of important information. Dickersin et al found that as much as 20% of relevant trials were published in languages other than English.4 Reviewers may think that non-English trials are not relevant because they are not properly reported, and are thus of poor quality, but there is no evidence that this is the case, and good evidence to the contrary. Moher et al compared the completeness of reporting, design characteristics, and analytical approaches of 133 randomised controlled trials published in English, French, German, Italian, and Spanish over the same period.8 No substantial differences by language were found for any item. Furthermore, a study of journals published in German found that German language trialists were substantially more likely to publish negative (or non-significant) findings in German language journals and positive (or significant) findings in English language journals (Egger M, unpublished data). As a result, systematic reviews that exclude non-English trial reports may overstate treatment effectiveness. In addition, other clinically relevant outcomes may be differentially reported in different countries; Hayashi and Walker compared the completeness of reporting of adverse effects in Japanese and American trial reports, and found that American trials tended to report a larger range of adverse effects, but in less detail, than their Japanese counterparts.9
The Cochrane Brain and Spinal Cord Injury Group
The Cochrane Brain and Spinal Cord Injury Group is a network of persons from all over the world, committed to assembling, maintaining, and disseminating high quality systematic reviews of the effectiveness of interventions in the prevention, treatment, and rehabilitation of brain and spinal cord injury. The aims and scope of the Group were first discussed at an exploratory meeting in London in September 1996. The Group was formally registered with the Cochrane Collaboration in February 1997. Work is now well under way to establish a specialised register of controlled trials relevant to the needs of the group, so that reviewers can readily access as much of the existing trial evidence as possible. A great deal has already been accomplished, largely thanks to the extensive hand searching efforts carried out by the London based Intensive Care National Audit and Research Centre. Over 50 journals and conference proceedings relevant to the prevention, treatment, and rehabilitation of brain and spinal cord injury have been hand searched by the Group including several major Japanese journals (table 1). However, there is much more to be done, and the hand searching of conference abstracts, non-Medline journals, journals published in languages other than English, and efforts to uncover unpublished studies remain major challenges. Although the Group has only recently been established, some systematic reviews have already been completed, and will be published in a forthcoming version of The Cochrane Library.10 These reviews include: the effectiveness of steroids in acute severe brain injury; steroids in acute spinal cord injury; the effectiveness and safety of anti-seizure prophylaxis following brain injury; treatment of aggressive behaviour after brain injury; vocational rehabilitation after head injury, and the effectiveness of pool fencing for the prevention of childhood drowning. As further evidence becomes available, either from new studies, or from the further identification of previous trials, these reviews will be updated to include this evidence. Several other systematic reviews are planned (table 2), but hundreds of potential review topics remain to be tackled, and enthusiastic reviewers are still in short supply.
If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.