Using high quality clinical databases to complement the results of randomised controlled trials: the case of recombinant human activated protein C
BMJ 2001; 323 doi: https://doi.org/10.1136/bmj.323.7318.923 (Published 20 October 2001) Cite this as: BMJ 2001;323:923
- Andrew Padkin, MRC training fellow in health services research (Andrew.Padkin{at}lshtm.ac.uk)a,
- Kathy Rowan, directorb,
- Nick Black, professor of health services researcha
- a Editorialby Hinds Health Services Research Unit, Department of Public Health and Policy, London School of Hygiene and Tropical Medicine, London WC1E 7HT
- b Intensive Care National Audit and Research Centre, Tavistock House, London WC1H 9HR
- Correspondence to: A Padkin
- Accepted 7 September 2001
Understanding the generalisability (or applicability) of the results of randomised controlled trials in typical clinical practice remains one of the key methodological challenges to achieving a more scientific basis for health care.1 Little effort is made to use a scientific approach to assess generalisability, document the use of a new intervention systematically, or determine whether the trials' results are replicated in real life, either in the original patient groups studied or in other patients who receive the intervention.
To explore these issues, we use a case study to describe the practical difficulties that exist for policymakers and clinicians in interpreting the results of a randomised controlled trial evaluating recombinant human activated protein C, a new drug for treating severe sepsis in intensive care patients. We suggest how an existing, high quality, clinical database could provide information on the generalisability of the trial results and on the likely financial consequences of the drug's introduction, and how it could be used to monitor the diffusion and effectiveness of the drug in typical clinical practice.
Summary points
Randomised controlled trials may be performed in atypical settings with atypical patients, making it difficult to assess the generalisability of the results
High quality clinical databases could be used to facilitate this assessment and to provide evidence of clinical effectiveness in typical clinical practice
The case for using a high quality clinical database in the assessment of recombinant human activated protein C, a new drug for treating severe sepsis in intensive care patients, provides an important topical example
Limitations of randomised controlled trials
Evidence based medicine has focused on understanding the factors affecting the internal validity of randomised controlled trials but has paid far less attention to their generalisability. This is reflected in the many instruments for assessing the quality of trials, which concentrate predominantly on identifying factors that may challenge internal validity.2 In many randomised controlled trials both the setting and the patients studied differ from those in typical clinical practice. So, even if the evaluated intervention provides significant benefit in the patients studied, whether and for whom it should be used in routine practice remains a matter of judgment. Information is rarely presented or available to help determine an appropriate policy.3 Consequently, the diffusion of a new intervention into routine practice is often haphazard, both within and outside the patient group in which the trial was performed. Even in the highly regulated arena of commercial pharmaceutical research there are many examples of drugs in common use outside their licensed indications. 4 5
New treatment for severe sepsis
Severe sepsis is common in intensive care and has a high associated mortality. Previous epidemiological studies have been hampered by the multiplicity of definitions used for severe sepsis and the diversity of patient groups studied. The search for a treatment for severe sepsis has been a high priority for many years, but, despite over 40 trials of inflammatory modulators, no effective intervention has been identified.
Recently, results from a major international randomised controlled trial (the PROWESS trial) of a potential agent for treating severe sepsis in intensive care were reported.6 The definitions used for severe sepsis (see box) were based on those developed by the American College of Chest Physicians and Society of Critical Care Medicine7 and covered a wide spectrum of illness severity. The trial was stopped after interim analysis showed that treatment with recombinant human activated protein C (also called drotrecogin alfa (activated)) was associated with lower 28 day mortality from all causes than placebo (24.7% v 30.8%). This translated into a number needed to treat of 16 to prevent one death by 28 days, considerably lower than the number needed to treat of 56 to prevent one death by 35 days for intravenous thrombolysis in acute myocardial infarction8—a widely accepted benchmark of effective clinical practice.
Definition of severe sepsis used in PROWESS trial6
Known or suspected infection
Plus
Three or more signs of systemic inflammation: Core temperature 38°C or 36°C Heart rate 90 beats/min Respiratory rate 20 breaths/min or arterial carbon dioxide tension (PaCO2) 32 mm Hg or mechanical ventilation for an acute process White cell count 12 000/mm 3 or 4000/mm 3
Plus
At least one sepsis induced organ dysfunction (organ dysfunctions may occur in cardiovascular, respiratory, renal, haematological, or metabolic systems)
As activated protein C heads towards the UK market, the National Institute for Clinical Excellence (NICE) is likely to be asked to decide whether the NHS should provide it and, if so, for which patients. If a decision has not been reached by the time the drug is marketed in early 2002 individual doctors will have to decide on the use of the drug for individual patients.
Possible policy responses
There are four possible policy responses to the PROWESS trial results: the results could be (a) rejected because they are not deemed valid, (b) rejected because they are not generalisable, (c) accepted as indicating use of the drug for all patients similar to those in the trial, or (d) accepted with more stringent indications as to who should receive the drug. We consider each of these options with respect to defining the further information necessary to make an informed decision and to the way in which a high quality clinical database could help with the decision.
Reject results as invalid
Although, as yet, there have been few challenges to the trial's internal validity, the question as to whether the 28 day survival advantage translates into improved long term survival without a substantive reduction in quality of life remains unanswered. Further follow up of the trial patients by the PROWESS investigators could provide an answer to this question. Without this information, the cost effectiveness of the drug remains uncertain, although it should be remembered that patients with severe sepsis are often young and that recovery from the acute episode is likely to be followed by many years of normal life.
Reject results as not generalisable
Can the trial results be generalised to intensive care practice in Britain? One of the trial's strengths is that it was pragmatic9: it did not standardise other interventions and so was designed to investigate the benefit of activated protein C in typical clinical practice. However, despite being conducted in 164 centres in 11 countries, the trial did not include any British centres, the reasons for which are unknown. The generalisability of the results depends on whether the patients studied in the trial can be considered representative of patients with severe sepsis in British intensive care units. This question has two components: whether patients with severe sepsis in British intensive care units are similar to those recruited into the trial, and whether aspects of intensive care other than the use of activated protein C are similar in Britain to those in the trial centres.
A first estimate of the generalisability of the trial results can be made by comparing the outcomes for the trial's control group with those for patients who satisfy the same criteria for severe sepsis in British intensive care units. Fortunately, Britain is one of the few countries in which it is possible to investigate these components of generalisability immediately, without having to collect relevant data prospectively. About 67% of adult general intensive care units in England, Wales, and Northern Ireland participate in a national comparative audit of patient outcome from intensive care, the case mix programme, coordinated by the Intensive Care National Audit and Research Centre (ICNARC). Data on patients from their first 24 hours in an intensive care unit and on their outcome at discharge from the unit and from hospital are collected according to precise rules and definitions and extensively validated before being included in the national database.10
Severe sepsis covers a wide range of illness severity, from single organ dysfunction (top) to multiple organ dysfunction (bottom). (Reproduced with subjects' permission)
Severe sepsis covers a wide range of illness severity, from single organ dysfunction (top) to multiple organ dysfunction (bottom). (Reproduced with subjects' permission)
Using this database, we found that 28% of all intensive care admissions between 1996 and 2000 met the definition for severe sepsis adapted from the PROWESS trial.11 Hospital mortality was 44.7%, far higher than the 30.8% mortality at 28 days seen in the trial's control group. This may be due to differences in the populations studied; differences in the baseline characteristics (demographic, diagnostic, illness severity or comorbidities); differences in the outcomes measured (hospital v 28 day mortality); or differences in other interventions used (apart from activated protein C). Using individual patient data from PROWESS control patients and patients from the case mix programme, adjustment could be made for these differences to provide an accurate comparison between the outcomes for the trial control patients and similar patients in British intensive care units.
Accept results and recommend drug for all similar patients
The third possible policy response is acceptance of the validity and generalisability of the PROWESS trial results with the recommendation that the drug be used for all patient groups studied in the trial. The financial implications of such a policy are considerable. The absolute number of patients who would be eligible for activated protein C is unknown, but if only half of the 28% of patients admitted to intensive care who met the definition for severe sepsis used in the PROWESS trial were eligible to receive the drug when the trial exclusion criteria were also considered, then 10 000 patients a year might be eligible in England and Wales alone. The manufacturers have not yet revealed the cost of the drug, but HA-1A (Centoxin), another drug marketed for treating severe sepsis in 1992, cost £2200 per treatment course.12 It is not unreasonable to suppose that activated protein C might cost between £3000 and £5000 per course, leading to a total potential cost of £30m-£50m a year in England and Wales, without including other patients who might also be eligible for treatment.
Accept results but set more stringent limits as to who should receive the drug
It might be suggested that use of the drug should be limited to those patients who would be expected to gain maximum benefit. Unfortunately, this fourth possible policy response would require data that are not available. The only source of such information would be subgroup analyses of the PROWESS trial. The dangers of such analyses are well recognised, and any results would have to be interpreted with great caution.13
The risks of limiting the use of activated protein C on the basis of subgroup analysis are compounded because the possibility of doing further large randomised controlled trials is fading. As current evidence suggests that treatment leads to the survival of one extra person for every 16 treated, it may be deemed unethical to deny activated protein C to control patients with similar characteristics to those in the PROWESS trial.
Using clinical databases to record typical clinical practice
In situations where a randomised controlled trial is not possible, a non-randomised approach should be considered.14 A well designed non-randomised study performed to coincide with the launch of activated protein C could not only monitor use of the drug but also provide information on how effective it is in typical clinical practice in predefined subgroups. This would not only supplement the subgroup analysis of data from the PROWESS trial but also permit investigation of those groups not included in the trial but for whom activated protein C will undoubtedly be used (such as patients under 18 years of age). Such a study would need to be large and to be started soon. This could be achieved by the intensive care units that contribute to the case mix programme because the data collection systems already exist and participating clinicians are familiar with the national aggregated database being used for research.15
Conclusions
Although this case study describes one particular drug, it illustrates a common problem in healthcare policymaking. Insufficient attention has been paid to the generalisability of the results of randomised controlled trials which, all too often, are conducted with patients and services that are atypical. Fortunately, a high quality clinical database exists for intensive care in England, Wales, and Northern Ireland. This can be used to test the generalisability of the information available on the effectiveness of activated protein C before an evidence based decision is made about its use. The case mix programme database could not only be used to aid the interpretation of PROWESS trial results but also to provide detailed evidence on the likely financial implications in typical clinical practice, to monitor the drug's diffusion into clinical practice, and to measure its clinical effectiveness in the real world. Integrating the results of a randomised controlled trial with those available from a high quality clinical database provides a powerful model for the assessment of healthcare interventions as they pass from trials into clinical practice.
Acknowledgments
We thank Dr J Nolan of the Royal United Hospital, Bath, for providing the clinical photographs. In addition to his work at the Health Services Research Unit, AP is a consultant in anaesthesia and intensive care medicine at the Royal United Hospital, Bath.
Contributors: This article arose out of discussions between all authors, and all participated in drafting and editing it. AP will act as guarantor.
Footnotes
-
Funding AP is funded by the Medical Research Council. KR is funded by the Intensive Care National Audit and Research Centre. NB is funded by the London School of Hygiene and Tropical Medicine.
-
Competing interests AP and KR have received travel and subsistence costs from Eli Lilly to attend conferences on critical care. The Intensive Care National Audit and Research Centre has undertaken analyses on severe sepsis as paid consultancy work for Eli Lilly, and all authors are involved with the centre: KR as director, NB as scientific advisor, and AP as clinical advisor.