Results

SAMPLE CHARACTERISTICS

In the hospital based study, 150 people were recruited and one person withdrew. In the postal survey, a total of 409 booklets (82%) were returned; 121 questionnaires were returned blank, of these 84 were considered ineligible to participate in the study (changed address; did not have multiple sclerosis; deceased). Therefore, the response rate was 69% (409–121/500–84). The characteristics of patients in the two samples were similar, a wide range of ages and disease duration was included, and the broad categories of disability were evenly represented (table 1).

ARE THE DATA HIGH QUALITY ?

For the hospital based sample there were no missing data, both types of SF-36 scores could be estimated for all participants, and results were not influenced by disease severity (table 2). In the postal survey, data quality was excellent in participants who reported that they walked unaided or with an aid. In wheelchair dependent participants, the proportion of missing data reached a maximum of 16.5% for two items (questions 3i, 3j) and, although scale scores could be computed for the vast majority of people, scores for summary measures could only be computed in 72% (all eight scale scores must be present for summary scores to be computed).

IS IT LEGITIMATE TO GENERATE SF-36 SCALE SCORES IN MULTIPLE SCLEROSIS ?

Results for the pooled sample are reported, but independent analyses of the hospital and postal samples yielded similar results. Although all response options were endorsed for each item, item response-option frequency distributions were quite symmetric for only five of the eight scales (BP, GH, VT, SF, RE). Response distributions were skewed towards less favourable health states (low scores) for the PF and RP scales, and skewed towards more favourable health states for the MH scale. Nevertheless, items within each scale had similar mean scores and standard deviations indicating that they were roughly parallel.

All item-own scale correlations, corrected for overlap, exceeded 0.40 indicating that the items in each scale measured a common underlying construct, and that the criterion of Ware et al of equivalence of item total correlations was satisfied. Only two item-own scale correlations did not exceed item-other scale correlations by more than two standard errors. Item 9a (“How much of the time did you feel full of life”) correlated similarly with the VT and SF scales (0.61 and 0.54 respectively), and item 9b (“How much of the time have you been a nervous person”) correlated similarly with the MH and RE scales (0.46 and 0.36 respectively). Therefore, these two items had a limited ability to discriminate between two constructs that are hypothesised to be distinct.

α Coefficients for all SF-36 scales exceeded 0.80 indicating that all scales generated reliable scores (table 3). Intercorrelations among scales (range 0.18 to 0.57) were substantially below their respective α values, indicating that they were measuring eight related but distinct constructs.

Scores for the eight SF-36 scales spanned the entire scale ranges and, therefore, demonstrated good variability. Scores for the PF and RP scales were positively skewed (skewness 1.13 and 1.32 respectively) indicating that respondents tended to be more physically disabled. Scores for the other six scales were more evenly distributed (skewness−0.50 to +0.40). There were notable floor or ceiling effects for four scales: RP (floor 61.6%); RE (floor 37.3%, ceiling 41.4%); PF (floor 23.4%); BP (ceiling 21.1%).

IS IT LEGITIMATE TO GENERATE SF-36 SUMMARY SCORES IN MULTIPLE SCLEROSIS ?

Results for the pooled sample are reported, but independent analyses of the hospital and postal samples yielded similar results. Principal components analysis of intercorrelations among SF-36 scales extracted only one component with an Eigenvalue greater than unity. Nevertheless, the Eigenvalue of the second component (0.996) was essentially unity and examination of the scree plot supported the hypothesis that a two dimensional model of health underpins the SF-36 in multiple sclerosis. However, these two components explained less than 60% of the total reliable variance in all SF-36 scales, and less than 75% of the reliable variable in five of the eight scales (table4), indicating that a substantial amount of information from SF-36 scales is lost when summary measures are reported in multiple sclerosis. In addition, the magnitude and pattern of scale to component correlations in multiple sclerosis differ from the United States general population, indicating that the scale weights used to generate scores for the summary measures are not entirely applicable to people with multiple sclerosis (table 4, fig 2).

• Download figure
• Open in new tab
• Download powerpoint

Figure 2

Plot of the SF-36 scale to component correlations. (A) SF-36 in the United States population (n=2.474). Prepared from data reported by Ware et al.20 (B) SF-36 in multiple sclerosis (n=438).

Discussion

This study has comprehensively examined the basic assumptions underpinning the use and scoring of the SF-36 in people with multiple sclerosis. High levels of data completeness indicate that the SF-36 is acceptable to patients in both hospital and community settings. However, data quality is somewhat compromised when the SF-36 is administered by postal survey to more disabled people. This finding probably represents physical limitations (for example, impaired vision and writing) rather than poor acceptance and understanding of the instrument because data quality in the more disabled hospital based subsample, where the instrument can be administered by interview if required, is excellent.

Results indicate that when using the SF-36 in multiple sclerosis, scale scores can be generated using Likert's method of summed ratings. All scaling assumptions were fully satisfied except item discriminant validity. There were two instances where items failed to achieve definite scaling successes. This is a minor problem because the other items in each scale fully satisfy the criterion and, therefore, anchor the construct being measured.33 More problematic are the floor and ceiling effects demonstrated for four scales. These exceed the recommended maximum of 15%,39 and represent subsamples of people for whom changes in health status may be underestimated or not detected by the SF-36.

The most important finding of this study is that scores for the two SF-36 summary measures should be reported with caution. Although PCA supports a two dimensional model of health, this model does not explain as much of the variance in SF-36 scales as required and the pattern of scale to component correlations is not entirely consistent with findings in other clinical populations.2 20 40 In particular, the BP scale correlates only moderately with the physical component (0.48), and the SF scale correlates only moderately with the mental component (0.51) and similarly with the physical component (0.60). Although small departures from predicted results are to be expected, and have been reported,2 these findings in multiple sclerosis seem to be more significant. The finding that the two components explain less than 60% of the variance in SF-36 scales suggests that even multiple sclerosis specific algorithms for summary scores, based on principal components analysis with orthogonal rotation, may not be feasible.

Other authors have raised concerns about the SF-36 summary scores because the weightings are based on factor score coefficients generated by orthogonal factor rotations.41-43 Simonet al argue that physical and mental health are related and, therefore, orthogonal rotations which assume factors are uncorrelated could generate misleading results. They support their argument with empirical evidence in primary care patients.41 Norvedt et al show that the SF-36 MCS underestimates the mental heath impact of multiple sclerosis,43 and that the RAND-36 physical and mental summary scores,42 the weightings of which are generated by oblique factor rotations (which assume the factors are correlated), provide mental health scores that are more consistent with multiple sclerosis.

The results of our study, however, suggest that the problem may be more fundamental than the method of factor rotation. Factor analysis is a data reduction technique that analyses the relations among a group of variables and identifies clusters of variables that are empirically distinct. Table 3 and figure 2 do not show distinct clusters among SF-36 scales. Indeed, correlations among the four scales hypothesised to correlate highest with the physical health component (PF, RP, BP, and GH) are notably lower (range 0.28 to 0.49; mean 0.36) than those reported by others (range 0.52 to 0.65; mean 0.5720) and similar to those between physical and mental scales (range 0.18 to 0.48; mean 0.37).

In view of the comments of Simon et al, and others who raise concerns about the use of principal components analysis or orthogonal factor rotations in health measurement,44 we repeated the factor analytical studies using different methods of extraction (principal axis, maximum likelihood, unweighted and generalised least squares) and oblique (promax) rotation. All methods of factor analysis generated similar results. Most scales (four or more) did not load uniquely on only one factor and had relatively high loadings (>0.50) on both factors. Furthermore, no clear factor solutions were generated. These findings are clinically sound. For example, fatigue can impact significantly on both physical and mental functioning and roles. Although our failure to replicate the SF-36 factor structure may represent cultural differences (United Kingdom versus United States), evidence suggests that this is not the case.45

Although this is a relatively small study the generalisability of these results is supported by two facts. Firstly, when tests of scaling assumptions underpinning the generation of SF-36 scale and summary scores were undertaken for the hospital and postal samples separately, similar findings were demonstrated. Secondly, SF-36 scale score distributions, floor and ceiling effects, and α coefficients reported in this study reproduce the findings of others.15 22Nevertheless, further studies are required to establish the replicability of these results and the interpretation of these findings. It is also important to note that this study has only considered data quality and scaling assumptions, criteria that should be satisfied before more detailed psychometric evaluations are undertaken. More extensive evaluations of SF-36 scales in people with multiple sclerosis are required to determine the extent to which they are valid and responsive indicators of the health constructs that they purport to measure.

This study did not consider the content validity of the SF-36 in multiple sclerosis. That is, to what extent do the items of the SF-36 compare with issues volunteered as important by patients in open ended interviews. Generic measures are designed to assess health domains thought to be universally relevant. Therefore, they have the advantage of enabling comparisons across diseases and interventions, but the disadvantage of failing to reflect domains and aspects of health that are disease unique. This may influence the potential of generic measures to detect change. These arguments underpin the development of disease specific health measures, and the use of both types of measures in studies.27

A limitation of our study was that we used an arbitrary criterion, the subjective judgment of one of the authors, to determine whether people in the hospital based sample had significant cognitive impairment precluding reliable completion of questionnaires. It would have been preferable to have had cognitive function formally evaluated, and a predefined empirically based cut off level for inclusion or exclusion. Unfortunately, such criteria do not exist as the relation between cognitive impairment and ability to complete self report questionnaires in a reliable and valid manner has not been formally evaluated. This is an important area for future research in patient based outcome measurement of neurological disorders associated with cognitive impairment.

The results of this study have implications for clinical trials in multiple sclerosis. Some new and expensive treatments are now available that have been shown to significantly affect the natural history of multiple sclerosis in terms of brain MRI and relapse rate but not impact on disability progression. As the disability measure used in all of these studies, the EDSS, has poor ability to distinguish between groups and individual patients and limited responsiveness,46 multiple sclerosis clinical trialists are looking for more rigorous methods of measurement of health status. There is, rightly, considerable interest in the SF-36. Furthermore, the benefits of SF-36 summary scores over scale scores (absence of floor and ceiling effects, better measurement precision, greater responsiveness, fewer statistical analyses) offer an attractive solution to many measurement problems. Results from this and other studies15 22 23 show floor and ceiling effects for SF-36 scales that may limit their usefulness in detecting treatment effects, and this study questions the validity of SF-36 summary scores.

This study also has wider implications for health measurement. The findings are of practical use for neurologists and other health professionals who may take measures “off the shelf” expecting that such attributes have been fully tested or that measures are generally applicable. The result might also be important in reporting accurately the strength of treatment or rehabilitation effects.

Conclusions

The findings of this study provide some empirically based guidelines for the use of the SF-36 in multiple sclerosis. High levels of data quality support its use as a health measure in diverse groups of people with multiple sclerosis. Tests of scaling assumptions suggest that SF-36 scale scores can be reported with confidence, but that summary scores are not valid. Results also suggest that use of the SF-36 might be limited to cross sectional studies as the floor and ceiling effects associated with scale scores could underestimate treatment effectiveness in clinical trials, and the extent of health changes in longitudinal studies. It is important to note that health measures such as the SF-36 are recommended for group comparison studies and not individual patient clinical decision making. This is because confidence intervals around individual scores are too wide to be able to make reliable and valid judgments at the level of the individual patient.39 Finally, this study provides further evidence of the need to evaluate psychometric properties, including scaling assumptions, in relevant clinical populations before these instruments are used to evaluate therapeutic effectiveness in disease specific samples.