Article Text

Download PDFPDF

Variation between hospitals in patient outcome after stroke is only partly explained by differences in quality of care: results from the Netherlands Stroke Survey
  1. H F Lingsma1,
  2. D W J Dippel2,
  3. S E Hoeks1,
  4. E W Steyerberg3,
  5. C L Franke4,
  6. R J van Oostenbrugge5,
  7. G de Jong6,
  8. M L Simoons1,
  9. W J M Scholte op Reimer1
  1. 1
    Thoraxcentre, Department of Cardiology, Erasmus Medical Centre, Rotterdam, The Netherlands
  2. 2
    Department of Neurology, Erasmus Medical Centre, Rotterdam, The Netherlands
  3. 3
    Department of Public Health, Erasmus Medical Centre, Rotterdam, The Netherlands
  4. 4
    Department of Neurology, Atrium Medical Centre, Heerlen, The Netherlands
  5. 5
    Department of Neurology, University Hospital Maastricht, Maastricht, The Netherlands
  6. 6
    Department of Neurology, Isala Clinics-Weezenlanden, Zwolle, The Netherlands
  1. Hester F Lingsma, MSc, Room AE-241, Erasmus MC, Postbus 2040, 3000 CA Rotterdam, The Netherlands; h.lingsma{at}erasmusmc.nl

Abstract

Background and purpose: Patient outcome is often used as an indicator of quality of hospital care. The aim of this study is to investigate whether there is a straightforward relationship between quality of care and outcome, and whether outcome measures could be used to assess quality of care after stroke.

Methods: In 10 centres in The Netherlands, 579 patients with acute stroke were prospectively and consecutively enrolled. Poor outcome was defined as a score on the modified Rankin scale ⩾3 at 1 year. Quality of care was assessed by relating diagnostic, therapeutic and preventive procedures to indication. Multiple logistic regression models were used to compare observed proportions of patients with poor outcome with expected proportions, after adjustment for patient characteristics and quality of care parameters.

Results: A total of 271 (47%) patients were dead or disabled at 1 year. Poor outcome varied across the centres from 29% to 78%. Large differences between centres were also observed in clinical characteristics, prognostic factors and quality of care. For example, between hospital quartiles based on outcome, age ⩾70 years varied from 50% to 65%, presence of vascular risk factors from 88% to 96%, intravenous fluids when indicated from 35% to 81%, and antihypertensive therapy when indicated from 60% to 85%. The largest part of variation in patient outcome between centres was explained by differences in patient characteristics (Akaike’s Information Criterion (AIC) = 134.0). Quality of care parameters explained a small part of the variation in patient outcome (AIC = 5.5).

Conclusions: Patient outcome after stroke varies largely between centres and is, for a substantial part, explained by differences in patient characteristics at time of hospital admission. Only a small part of the hospital variation in patient outcome is related to differences in quality of care. Unadjusted proportions of poor outcome after stroke are not valid as indicators of quality of care.

Statistics from Altmetric.com

Request Permissions

If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.

Assessment of quality of care is becoming more and more important in medical practice. Donabedian has argued that quality in health care can be viewed as a function of three components: structure, process and outcome.1 The first scientific forum of the American Heart Association and the American College of Cardiology on assessment of health care quality in cardiovascular disease and stroke has elaborated this framework for stroke care.2 They proposed a series of possible performance measures in the three different domains, based on the existing guidelines for stroke.

Despite this suggestion, quality of care in stroke is still often evaluated by use of outcome measures, usually (standardised) mortality rates on hospital level.3

Outcome assessment is generally easier than assessment of process measures, and it is often assumed that outcome measures reflect the relative importance of the different aspects of the care process, which makes them most relevant for patients. Those in favour of process indicators often express doubt about whether outcome really reflects quality of care as outcome largely depends on patient characteristics. Furthermore, flaws in care and well-performed care may cancel each other out, and may not be reflected in overall outcome. This problem of outweighing good and bad performance becomes even larger in analyses at hospital level, as these also use the average quality scores of individual patients.3 4

Several studies have investigated the validity and feasibility of outcome data as indicators of quality of stroke care, with diverging results and conclusions, mostly due to limited information on the quality of care process.58 For this study, data were derived from the Netherlands Stroke Survey, in which detailed data on both patient characteristics and process of care are available. Therefore, this survey offers a unique opportunity to investigate whether there is a straightforward relationship between quality of care and outcome, and whether outcome measures could be used to assess quality of care after stroke.

METHODS

Study population

The Netherlands Stroke Survey was conducted in 10 centres in The Netherlands: 2 in the North, 4 in the Middle and 4 in the Southern regions. The participating sites comprised 1 small (<400 beds), 4 intermediate (400–800 beds) and 5 large centres (>800 beds). Two centres were University hospitals. All centres had a neurology department, a neurologist with expertise in stroke and a multidisciplinary stroke team. All but one hospital had a stroke unit, 8 were participating in a regional stroke service and 9 were equipped for thrombolytic therapy. These institutions deliver care to approximately 10% of all acute stroke patients in The Netherlands, and their size and stroke expertise can be considered representative of hospital-based stroke care in The Netherlands.9

All patients who were admitted to the neurology department with suspected acute stroke between October 2002 and May 2003 were screened. Patients were enrolled consecutively and prospectively if the initial diagnosis of first or recurrent acute brain ischaemia was confirmed by the neurologist’s assessment and if symptom onset was less then 6 months ago. All patients were admitted to the neurology department and were followed throughout their hospital stay. All patients or their proxies provided informed consent and the Medical Ethics Committees and Review Boards of the participating hospitals approved the study.

Data collection

Trained research assistants collected data from the patients’ hospital charts, within 5 days after discharge. At 1 year, survival status was obtained through the Civil Registries. In all survivors, a telephonic interview was conducted based on a structured questionnaire, which was sent in advance. Follow-up was complete in 96% of the patients. More details on the study population and methods of data collection can be found in an earlier publication of this survey.9

Clinical characteristics and prognostic factors

Stroke subtype (brain infarction, haemorrhagic brain infarction, transient ischaemic attack or amaurosis fugax) was defined by the treating neurologist based on clinical features and brain imaging (computed tomography (CT) or magnetic resonance imaging (MRI)) data. Previous stroke was defined as present if ischaemia of brain or eye or cerebral haemorrhage was noted in the medical history. Level of consciousness was assessed with the Glasgow Coma Scale,10 and disability in activities of daily living with the Barthel Index.11 Atrial fibrillation and ischaemic heart disease were marked if diagnosed by physical examination or if detected on ECG or if noted in the patient’s medical history. In addition, peripheral vascular disease, diabetes mellitus and hypertension were based on the patient’s medical history, or scored if diagnosed during hospitalisation. Hyperlipidaemia was defined as present if total serum cholesterol exceeded 5 mmol/L, or if low-density lipoprotein exceeded 3.2 mmol/L or if hyperlipidaemia was noted in the patient’s medical history. The presence of carotid stenosis ⩾70% was assessed by carotid imaging.

Quality of care

To measure quality, we distinguished between acute stroke treatment, sub-acute stroke care and prevention. Quality of care parameters in acute stroke treatment involved the use of CT or MRI, electrocardiogram (ECG), appropriate laboratory tests, the administration of acetylsalicylic acid within 48 hours and thrombolytic therapy within 3 hours. Sub-acute care included the administration of intravenous fluids, swallowing test, percutaneous endoscopic gastrostomy tube (PEG tube) insertion when indicated, early mobilisation and early physiotherapy. Prevention included assessment of risk factors, measurement of serum cholesterol, carotid endarterectomy within 6 months, antiplatelet therapy, oral anticoagulants, antihypertensive therapy and cholesterol-lowering therapy. These quality of care parameters and their indications were selected from national guidelines and most of them are also mentioned in the report from the American Heart Association/American College of Cardiology on assessment of healthcare quality in cardiovascular disease and stroke.2 Each parameter was considered to be present in a certain patient when the diagnostic or therapeutic procedure was carried out and was indicated, or was not carried out and was not indicated. Otherwise, the indicator was considered to be absent. The quality of care parameters have been described more extensively in an earlier publication of this survey.9

Outcome measures

Poor outcome was defined as dead or disabled at 1 year—that is, a score on the modified Rankin scale ⩾3.12 Additional outcome measures were dead or disabled at discharge, 30-day mortality and 1-year mortality.

Statistical analyses

To assess differences between centres in clinical characteristics, prognostic factors, quality of care parameters and outcome measures, centres were grouped in quartiles based on the percentage of patients dead or disabled after 1 year. These quartiles were fixed for all further analyses. P-values were derived from Chi square tests for differences between the 10 centres.

We performed stepwise logistic regression analysis with backward elimination of predictors to construct prediction models for poor outcome. The selection criterion for inclusion was p<0.157.13 In step 1, only clinical characteristics (age, sex and duration of symptoms) were entered into the model. In step 2, patient-related prognostic factors were added: stroke severity, consciousness level at hospital arrival, Barthel Index at hospital arrival, previous stroke, atrial fibrillation, history of ischaemic heart disease, peripheral vascular disease, diabetes mellitus, hypertension, hyperlipidaemia, admission glucose ⩾11 mmol/L and independent pre-stroke living arrangement. In step 3, the mentioned quality of care parameters were added to the model. The main interest was not the relationship of individual predictors with outcome but the predictive strength of the different steps (clinical characteristics, other patient-related factors and quality of care). The contribution of each step was expressed by Akaike’s Information Criterion (AIC), which corresponds to the χ2 of the step (or the difference in –2 log likelihood between the model with and without that step) minus 2× the degrees of freedom.13

The discriminative ability of the model was expressed by the area under the receiver operating characteristic (ROC) curve. This area represents the probability that, within pairs of one patient with and one without the outcome, the patient with the higher prediction actually had the outcome.14

We calculated w scores to estimate the absolute differences in the number of patients with poor outcome between centres, before and after adjustment for clinical patient characteristics, prognostic factors and quality of care parameters.5 The w score of a hospital expresses the difference between the observed and predicted number with poor outcome per 100 patients and is calculated by the formula [(o-p)/n]*100, where o is the observed number of patients with poor outcome, p the predicted number of patients with poor outcome and n is the number of patients. For the unadjusted w scores, we derived p at each hospital by multiplying the number of patients (n) by the proportion of patients with poor outcome in the total population. For the adjusted w scores, we derived p at each hospital by summing the individual predicted probabilities generated by the logistic regression models.6 Calculations of 95% CIs for the w scores were made using the method described by Parry et al.15 The total variation between centres was also expressed as the percentage of patients with a different outcome than expected. We performed all analyses using SPSS 13.0 for Windows and Microsoft Excel.

Sensitivity analyses

We repeated the logistic regression analysis for three alternative outcome measures: dead or disabled at discharge, 30-day mortality and 1-year mortality. Furthermore, we repeated the logistic regression analysis by modelling the three steps (clinical characteristics, prognostic factors and quality of care parameters) in different orders.

RESULTS

Outcome

The study population consisted of 579 patients who were admitted to the hospital due to stroke. Of all patients, 59 (10%) died during the hospital stay. Of the remaining 520 patients, 206 (39%) were disabled at discharge. At 1 year, 143 patients (25%) were dead and 128 of the remaining 436 patients (29%) were disabled (modified Rankin scale 3, 4 or 5). So, the total number of patients with poor outcome at 1 year after stroke was 271 (47%). This percentage increased from 37% in hospital quartile 1 to 75% in hospital quartile 4 (fig 1). For all outcome measures (mortality, disability and composite), both during the short term (discharge and 30 days) and at 1 year, we observed the same trend across the hospital quartiles.

Figure 1 Outcome after ischaemic stroke by hospital and quartile division (n = 579).

Clinical characteristics and prognostic factors

Of all patients (n = 579), 90% were admitted within 48 hours after symptom onset, and 95% within 1 week. Mean age was 70.4 (±13.2), 311 patients (54%) were male, the majority of patients (510, 88%) were diagnosed with brain infarction and 536 patients (93%) had one or more vascular risk factors. Regarding the symptoms of stroke, 13% of the patients had a lowered consciousness level and 89% were ADL (activities of daily living)-dependent at hospital admission (table 1).

Table 1 Variation in clinical characteristics and prognostic factors by hospital

A number of differences in relative frequency of patient characteristics between the hospital quartiles was observed. Some were moderate (for example, the presence of vascular risk factors) and some were large (for example, age ⩾70 years, lowered consciousness level and ADL dependency at hospital admission) (table 1).

Quality of care

The majority of the patients received the recommended diagnostic investigations and medical treatment in the acute phase, with the exception of thrombolytic therapy. Performance of a 12-lead ECG, provision of acetylsalicylic acid within 48 hours and thrombolytic therapy differed between the centres (p = 0.045, p<0.001 and p<0.001, respectively); performance of CT/MRI and laboratory tests did not (p = 0.494 and p = 0.624, respectively) (table 2).

Table 2 Variation in acute management of ischaemic stroke by hospital

Sub-acute care was less often performed in adherence to national guidelines. Of all 510 patients with a brain infarction, 203 (40%) underwent a swallowing test, 121 (24%) were mobilised on the first day and 106 (21%) had physiotherapy during the first day. Of the 413 patients with brain infarction and no parenteral feeding, 198 (48%) received intravenous fluids. For all sub-acute process measures, differences between centres were observed (p<0.001), with the exception of PEG tube insertion.

Performance of secondary prevention also varied considerably between centres. The proportion of patients that underwent carotid imaging when indicated varied between 33% and 92% across the centres’ quartiles (p<0.001). Only 9 of 52 patients (17%) with carotid stenosis ⩾70% underwent carotid endarterectomy within 6 months. The number of patients without atrial fibrillation that received antiplatelet therapy was high (93%), but there was still a significant difference between the centres (p<0.001). The proportion of patients that received oral anticoagulants and antihypertensive therapy when indicated also differed across centres (p = 0.048 and p = 0.029), whereas laboratory tests and cholesterol-lowering therapy in patients with indication did not (p = 0.304 and p = 0.085, respectively) (table 3).

Table 3 Variation in secondary prevention after ischaemic stroke by hospital

Relationship between clinical characteristics, prognostic factors, quality of care and outcome

Predictive factors in the model were age, sex, duration of symptoms, severe stroke, lowered consciousness level at hospital arrival, Barthel Index at hospital arrival, previous stroke, atrial fibrillation, ischaemic heart disease, diabetes mellitus, hypertension, hyperlipidaemia, ECG performed, mobilisation on day 1, antiplatelet therapy and oral anticoagulation.

Age, sex and duration of symptoms explained a large part of the variation (AIC = 54.7, p<0.001) and another substantial part was explained by prognostic factors (AIC = 79.3, p<0.001). Quality of care explained a relatively small part of the variation (AIC = 5.5, p = 0.009). The area under the curve of the model with only patient characteristics was 0.80 and that of the complete model 0.82, indicating a reasonable predictive performance (table 4).

Table 4 Multivariate analysis: predictors of outcome (dead or disabled at 1 year) after ischaemic stroke

w scores

The differences in outcome between centres were also expressed in w scores. Before any adjustment, the sum of the absolute w scores across the ten centres was 142, indicating that, over all of the centres, 14.2% of the patients had a different outcome (better or worse) than expected. After adjustment for age, sex and duration of symptoms, this percentage was reduced to 11.2%. After further adjustment for prognostic factors, the total variation declined further to 9.5%. After adjustment for quality of care, 8.8% of the patients still had a different outcome then expected, which could not be explained by the variables taken into account in this study (fig 2).

Figure 2 Differences in observed number of patients with poor outcome and predicted number (w score) per hospital. Number of observed dead or disabled at 1 year per 100 patients above or below number predicted.

Sensitivity analyses

Results were not affected by changing the dependent variable of the logistic regression into dead or disabled at discharge, 30-day mortality or 1-year mortality. The backward elimination of predictors resulted in slightly different predictors remaining in the model, but the different steps had approximately the same predictive power as in the initial model. Changing the order of the steps did not affect the results; the predictive strength of patient characteristics remained much larger than that of quality of care parameters.

DISCUSSION

We explored the validity of patient outcome as an indicator of differences in quality of care between centres. We compared the observed with the expected outcome in 10 representative centres and we investigated whether differences in patient characteristics and quality of care could explain differences in outcome between the centres. We found that clinical characteristics and prognostic factors explain a relatively large part of the variation in outcome, whereas quality of care parameters explain a much smaller part.

In addition, previous studies observed considerable variation between centres in outcome after stroke and were unable to explain this variation with differences in quality of care.58 16 The strength of our study is the detailed data on quality of care parameters that have been considered important in evidence-based guidelines. Despite this, we were also not able to demonstrate a clear and consistent relationship between quality of care and outcome on top of patient characteristics. Our sample size was small, however, and we may have had insufficient power to detect small effects.

Data were collected in 2002 and 2003. Improvements in the process of care might have led to a stronger relationship with outcome. However, since the time of the study, no major changes in the care process have taken place.

An explanation for our results that evidence- and consensus-based measurements of the process of care appear to have such little impact on outcome, could be that treatment effects are generally modest. This is reflected in the fact that large randomised controlled trials are needed to identify a benefit of treatment. In quality of care studies, however, we are looking for differential use of established treatments and the resulting differences in outcome will therefore be even smaller. Also, not all items of care or treatments apply to all patients and so cannot be expected to have a large impact on aggregated outcomes made up of all patients.

It should be noted that we defined poor patient outcome as Rankin scale ⩾3 and it could be questioned whether it is justified to put patients with Rankin scale 3 and patients who died into the same category. On the other hand, sensitivity analysis with 1-year mortality as outcome did not change the results. Furthermore, pre-stroke modified Rankin scale could have explained part of the variation in outcome, but we could not adjust for it as it was not available in our dataset. We did, however, adjust for previous stroke and independent pre-stroke living arrangement, as a proxy for pre-stroke functional status.

Variation in stroke patients’ outcome between centres was determined more by clinical characteristics and prognostic factors than by hospital variation in quality of care. It would therefore make more sense to monitor the process of care directly in order to assess quality of care. There are several examples where this is being done—for example, the RIKS stroke register from Sweden;17 the National Sentinel Audit from England, Wales and Northern Ireland;18 and the Scottish Stroke Care Audit.19 In The Netherlands, however, unadjusted 7-day mortality rates were published on the internet until 2006.20 A clear advantage of measuring process parameters instead of outcome is that it directly identifies opportunities for improvement in all hospitals, not only in those with poor outcome. This approach has successfully been applied in England and Wales in the form of regular national audits of stroke care using the Intercollegiate Stroke Audit Package.21

Those in favour of outcome assessment advocate that quality assessment on process level requests a too-detailed data collection, and conclusions on quality depend largely on the selection of process measures.3 Our study shows, on the other hand, that using outcome assessment for quality measurement is only valid after adjustment for patient characteristics. This approach is increasingly adopted (eg, in the UK) where total hospital mortality rates are adjusted for some key patient characteristics22 and in the USA where adjusted mortality rates after acute myocardial infarction and heart failure are used to compare centres.23 24 However, even if complete adjustment for patient characteristics is possible, this may not be sufficient for a meaningful comparison of outcomes between centres. Centres with patients with a good prognosis, small deficits and less comorbidity may still be more likely to deliver good quality of care compared with centres with more complex patients. The former ones have fewer patients with an indication for certain interventions, and hence they are less likely to withhold these interventions. For example, a patient without swallowing problems does not need a PEG tube, so it cannot be withheld unjustly. There are simply less opportunities to deliver substandard care. This implies that adjustment for patient characteristics may also be necessary when process measures are used.

Recently, attention is given to the development of prognostic models for outcome after stroke, which may be useful for quality assessment through proper outcome adjustment.2530 These models should be validated, however, in databases from the concerning country and they should be updated regularly. An important question is also whether a model is feasible in the sense that it can be fed by routinely collected data. As well as the discussion on which patient characteristics should be included in models for adjustment, the feasibility and validity of different methodological and statistical approaches should also be investigated and discussed.

Part of the variation in outcome remained unexplained. It might be that we failed to measure important aspects of care—for example, how well complications are identified and treated. However, it seems implausible that such aspects of care are likely to have a huge impact on outcome. Another explanation is that there might be differences in patient characteristics that we cannot quantify. It remains unexplained how large residual variation in outcomes remains after adjusting for all known factors. More research is needed to clarify this phenomenon.

We conclude that patient outcome largely varies between centres and is, to a substantial degree, explained by differences in patient characteristics at the time of hospital admission. Only a small part of the hospital variety in patient outcome is related to differences in the quality of care process. Unadjusted proportions of poor outcome after stroke are not valid as indicators of quality of care.

Acknowledgments

Participating centres: Albert Schweizer Kliniek, Dordrecht, Dr RP Kleyweg; Atrium Medisch Centrum, Atrium Heerlen, Atrium Brunssum, Atrium Kerkrade, Dr CL Franke; Erasmus MC, Rotterdam, Dr DWJ Dippel; Ziekenhuis St. Jansdal, Harderwijk, Dr DJ Hofstee; ISALA Klinieken, Ziekenhuis De Weezenlanden, Ziekenhuis Sophia, Zwolle, Dr G de Jong, Dr JSP van den Berg; Laurentius Ziekenhuis, Roermond, Dr A Schyns-Soeterboek; Medisch Centrum Rijnmond-Zuid, Locatie Clara, Locatie Zuider, Rotterdam, Dr R Saxena, Dr E Thomeer; St. Franciscus Gasthuis, Rotterdam, Dr FH Vermeij; TweeSteden Ziekenhuis, Tilburg, Dr BPW Jansen; St Elisabeth Ziekenhuis, Tilburg, Dr PLM de Kort; Academisch Ziekenhuis Maastricht, Maastricht, Dr RJ van Oostenbrugge.

Data collection officers: M Beijerink, H Broers, A van den Dool, H Fransen, Chr Jansen, M Kamps, C Koppelaar, G Middelveld-van Berkum, CW van Rijn, EAM de Swart, HAM Tebbe, M van der Wal, M Vermeulen, S Versluis.

REFERENCES

Footnotes

  • See Editorial Commentary, p 852

  • Funding: This work was funded by the Netherlands Heart Foundation (2000T101).

  • Competing interests: None declared.

  • Ethics approval: Ethics approval was obtained.

Linked Articles