Article Text

PDF

The MS Symptom and Impact Diary (MSSID): psychometric evaluation of a new instrument to measure the day to day impact of multiple sclerosis
  1. J Greenhalgh1,
  2. H Ford2,
  3. A F Long1,
  4. K Hurst3
  1. 1Health Care Practice R&D Unit, University of Salford, Allerton Building, Frederick Road, Salford M6 4WT, UK
  2. 2United Leeds Teaching Hospital’s Trust, Beckett Street, Leeds LS9 7TF, UK
  3. 3Nuffield Institute for Health, University of Leeds, 71–75 Clarendon Road, Leeds LS2 9PL, UK

    Abstract

    Objectives: This study aimed to develop further a diary originally devised to measure the impact of multiple sclerosis (MS) as part of a cost utility study of beta interferon, and to evaluate its reliability, validity, and responsiveness in an outpatient sample of people with MS.

    Methods: The original diary was further developed using qualitative and quantitative methods to ensure that it addressed the views of people with MS. The psychometric properties of the MS Symptom and Impact Diary (MSSID) were evaluated in a sample of 77 people who completed the MSSID daily for 12 weeks. Internal and test–retest reliability, discriminant and convergent validity, and responsiveness were assessed using traditional psychometric methods.

    Results: The MSSID formed three, internally consistent scales that measured mobility, fatigue, and the overall impact of MS. The test–retest reliability of the mobility scale was adequate for individual comparisons (ICC>0.90) and the fatigue and overall impact scales were adequate for group comparisons (ICC>0.70). The MSSID was able to distinguish between clinical groups depending on clinical course, indoor ambulation status, and relapse status. It demonstrated associations with other single point instruments in the expected direction. Compared with single point instruments, its responsiveness was similar or better, especially in detecting short term improvements in functioning.

    Conclusions: The MSSID may provide a useful complement to currently available instruments to measure the outcomes of MS within clinical trials. Further research is needed to explore its feasibility in the context of a randomised controlled trial and its utility for clinicians.

    • multiple sclerosis
    • psychometrics
    • diary
    • BI, Barthel Index
    • MS, multiple sclerosis
    • LHS, London Handicap Scale
    • Leeds MS quality of life scale
    • MSIS-29, MS Symptom and Impact Scale-29
    • MSSID, MS Symptom and Impact Diary

    Statistics from Altmetric.com

    The importance of measuring outcomes from the perspective of people with multiple sclerosis (MS) has recently become more widely recognised.1,2 People with MS have different concerns to clinicians regarding the most important areas in which MS affects their lives; clinicians place more importance on the physical aspects of the condition, while patients are more concerned with mental health and vitality.3 The relationship between clinical assessments of disability and those of the patient have also typically been low.4,5 Moreover, disease modifying treatments for MS, such as beta interferon, may produce benefits that are significant to the patient but are not measured by clinical assessments.6

    The content of outcome measures needs to reflect the issues that people with MS consider to be important.7 It is only recently that disease specific instruments to measure the outcomes of MS have been developed using a patient centred approach. These include instruments such as the Leeds MS quality of life scale (LMSQoL)8 and the MS Symptom and Impact Scale-29 (MSIS-29)9,10 both of which have been shown to be reliable, valid, and responsive to change.

    Such instruments represent a significant contribution to the measurement of outcomes from the perspective of people with MS. However, they are not equipped to measure a central feature of MS as a disease that has a significant effect on the lives of people with MS, namely the day to day variability of symptoms and their impact.11,12 The variable and unpredictable nature of MS makes it difficult for people with MS to plan their lives or be spontaneous, and results in increased levels of psychological distress,13,14 but is rarely measured in clinical trials or in clinical practice. The ambiguous and transient nature of many MS symptoms can also make it difficult for people with MS and for clinicians to identify when a relapse has occurred and when recovery begins.15,16

    There is a need for new ways of assessing the outcomes of MS that take account of its variable nature. Increased importance should be placed on the use of symptom diaries in trials to accurately document the onset and resolution of relapses and symptoms.15 Diaries have been frequently used as outcome measures in randomised controlled trials of chronic diseases.17–19 They can provide a more comprehensive view of people’s health problems and enable the investigation of short and long term changes in health patterns over time.18–20 They have also been used to measure the quality of life of people with MS more comprehensively within a study of the cost utility of beta interferon, but as yet, the psychometric properties of this diary have not been tested.21 This paper reports on a study that modified and extended this diary,21 with reference to the views of people with MS, and assessed its validity, reliability, and responsiveness in an outpatient sample of people with MS using traditional psychometric methods.23,24

    METHODS

    Development of the MSSID

    The MSSID was developed further in three phases. Firstly, two focus groups (n = 8 and n = 5) and ten in depth interviews with a purposively selected sample of people with MS, recruited from a community based population, explored the impact of MS and the relevance of diaries. Their views were used to modify the original diary.21 Secondly, the modified diary was pre-tested with nine people with MS using "think aloud" interviews,25 in which respondents gave a verbal report of their thought processes as they completed the diary to identify difficulties with the phrasing or response options of the items. Thirdly, the refined diary was piloted with 14 people with MS recruited from outpatient clinics, who completed the diary for 14 consecutive days. The final version of the MSSID (see appendix 1), in the form of an A5 booklet, comprised eight items. Two items (Q1 and Q2) asked about the experience and impact of 14 MS related problems on daily activities, four items (Q4–Q6) asked about the overall impact of MS, one item (Q7) compared health today with health yesterday, and the final item (Q8) left space for respondents to note down anything else that they considered to be important. This version of the MSSID formed the focus for psychometric assessment.

    Participants

    The psychometric properties of the MSSID were assessed in a sample of 77 people with MS recruited from outpatient clinics in two centres, between September and October 2000. People with MS attending the clinic were eligible if they were diagnosed with MS, aware of their diagnosis, and aged between 18 and 65 years. Those unable to complete a questionnaire owing to cognitive impairment were excluded. Those who had been diagnosed with MS for less than 1 year and those who were judged by the clinician to be experiencing difficulties dealing with their diagnosis were also excluded, to avoid the possibility of causing undue distress. The study was approved by the relevant local research ethics committees, and all participants gave written informed consent before taking part.

    Instruments

    At recruitment, the mobility status of all participants was rated according to whether they moved around the clinic unaided or used a walking aid. Participants were asked to complete three consecutive, 4 week diaries. At baseline and at the end of the 12 week study, participants completed a postal questionnaire. This included the LMSQoL8 as a measure of quality of life; the MSQoL-5427 as a measure of health status (the sexual functioning items were excluded as they have been found to be poorly completed26 and did not reflect the content of the diary); the postal Barthel Index28 as a measure of disability/functioning; and the London Handicap Scale (LHS)29 and the Reintegration to Normal Living Index (RNLI)30 as measures of handicap or participation. At the end of each 4 week diary period, participants completed a questionnaire asking about changes to symptoms and MS overall, medication, and relapses using questions from the cost utility study.21

    Data quality

    Data quality was assessed by examining response rates, missing items, and score distributions.

    Reliability

    Both internal and test−retest reliability were assessed. The internal structure of the MSSID was examined using principal components analysis with varimax rotation on three randomly selected days, one from each 4 week diary period. Two sets of analyses were performed, one for the symptom items (Q2) and one for the impact items (Q3–Q6). Factors were entered into the rotation if they had an eigenvalue of 1 or more, and loadings of 0.4 and above were used to determine which items belonged to which factor.31 The item content of the resulting factors were independently reviewed by a consultant neurologist (HF) to ensure they made clinical sense. Factor scores for each day were calculated by summing the items in each factor and then dividing by the total number of items completed within the factor that day. Higher scores indicated better functioning.

    The internal reliability of the resulting factors was assessed using Cronbach’s alpha32 and item total correlations on three randomly selected days, one from each month. Reliability was deemed adequate if Cronbach’s alphas were >0.824 and item total correlations >0.2.23

    The test−retest reliability of the resulting factors was assessed in two ways using the intraclass correlation coefficient (ICC).33 Firstly, the ICC was calculated between median week 1 and week 4 factor scores in stable respondents—that is, those whose symptoms did not change and who did not have a relapse during the first 4 weeks (n = 26). Secondly, the ICC was calculated between factor scores on the first 2 days of each 4 week diary period when the respondent indicated that their health had not changed (n = 61). ICCs between 0.70 and 0.89 were deemed adequate for group comparisons, while ICCs of 0.9 and above were deemed adequate for individual comparisons.24

    Validity

    Both discriminant and convergent construct validity were assessed using appropriate summary indicators of the longitudinal diary data.34 To provide a summary indicator of the impact of symptoms on daily activities (factors 1 and 2) and the impact of MS overall (factor 3), median factor scores were used. The median was chosen rather than the mean as it was expected that the median would be less influenced by missing data, and it was not expected that an individual’s scores over time would be normally distributed.

    Previous research has demonstrated that people with progressive MS experience greater levels of impairment and disability than other disease course groups,35 and that people who have had a relapse experience worse health status than those who have not.21 Therefore, it was hypothesised that there would be statistically significant differences in median factor scores between those with relapsing remitting or benign MS v those with progressive MS, between those who walked unaided v those who used a walking aid and between those who experienced a relapse during the study v those who did not. In each case it was expected that for patients with progressive MS, those who used a walking aid and those who had a relapse would have worse (lower) median factor scores.

    The convergent validity of the MSSID was examined by exploring correlations between the median factor scores during the last 4 week diary period and single point instruments collected at the end of the study. Correlations <0.4 were considered small, between 0.41 and 0.7 moderate, and 0.71 and above high.9

    Responsiveness

    Effect sizes of the change in median MSSID scores, LMSQoL, and MSQol-54 scores between week 1 and week 4 were calculated separately for those who indicated that their MS had got better (n = 14) and those who indicated that their MS had got worse (n = 20) in the first 4 weeks of the study.36 Effect sizes of 0.2 were considered to be small, 0.5 moderate, and 0.8 or more large.37 It was expected that effect sizes for the MSSID scores and MSQol-54 would be positive for those who got better and negative for those who got worse. The reverse pattern was expected for the LMSQoL because higher scores indicate worse functioning.

    Statistical analyses

    The differences in continuous variables (for example, MSSID scores) were assessed using Mann-Whitney U tests. Differences in categorical variables were assessed using the χ2 test. Relationships between MSSID factor scores and single point instruments were assessed using Spearman’s rank correlations. All analyses were carried out in SPSS for Windows (version 9).

    RESULTS

    Changes made to the original diary

    A number of changes were made to the original diary following the developmental stages. Seven symptom items were added to reflect issues that were raised in focus groups and interviews (for example "difficulties thinking clearly or forgetting things" and "feeling frustrated"). Some items were reworded to make them more interpretable to respondents (for example "altered sensations" was changed to "altered or loss of sensation"). The number of response options of some items was increased to better reflect the variation in symptom impact and health perceptions (for example, the response options for Q6 were reworded and increased from three to six). Finally, an open ended question (Q8) was added to allow respondents to record any additional issues or events that happened.

    Sample characteristics

    Sixty three people (82%) returned all three diaries; these were classed as respondents and used in subsequent analyses. Eighty one percent were women, the average age was 46 years, and they had been diagnosed with MS for an average of 11 years (table 1). Seventy three percent were currently not in paid work and 33% had experienced a relapse during the study. There were no statistically significant differences in age, gender, length of time since diagnosis, or clinical course between respondents recruited from either centre, thus the two samples were analysed together.

    Table 1

    Clinical and demographic characteristics of the sample

    Data quality

    Across the entire 12 weeks, 7% of MSSID items were missing, ranging from 5.8% to 7.6% across items. Three quarters of respondents completed at least 93% of the items and all respondents completed at least 50% of the items. Three symptom impact items (Q2) "weakness or heaviness in your legs", "problems with your bladder and bowel", and "feeling frustrated", had floor effects, with 20% or more person days rated as "a lot".

    Internal reliability

    The factor analysis of symptom impact items (Q2) revealed three factors that explained between 62% and 68% of the total variance. However, one factor contained only two items ("blurred, double, or shaky vision" and "coughing or choking when eating and drinking") and performed poorly in the internal reliability analyses, thus it was not retained. The remaining two symptom impact factors measured "mobility" and "fatigue". The factor analysis of the overall impact items (Q3–Q6) formed one factor that explained between 66% and 79% of the total variance and measured the overall impact of MS. For the three retained factors, Cronbach’s alpha exceeded or approached 0.8 and item total correlations exceeded 0.2 (table 2).

    Table 2

    Internal and test–retest reliability analysis of the MSSID

    Test–retest reliability

    For both weekly factor median scores and daily factor score comparisons, the ICC for fatigue and overall impact exceeded 0.7 and for mobility, exceeded 0.9 (table 2).

    Discriminant validity

    There were statistically significant differences in the overall impact scores between all the clinical groups in the expected direction (table 3). For mobility, there were statistically significant differences between all clinical groups in the expected direction with the exception of the relapse/no relapse comparison. For fatigue there were statistically significant differences between all clinical groups in the expected direction with the exception of the relapsing remitting/benign v progressive MS comparison. There were no statistically significant differences in the median factor scores between those recruited from either centre (data available on request).

    Table 3

    Discriminant validity analysis of the MSSID

    Convergent validity

    Table 4 shows the Spearman’s rank correlation coefficients between median factor scores during the last four weeks and the single point instruments collected at the end of the study. Mobility showed moderate correlations with the MSQol-54 physical and social functioning scales and with the RNLI. Fatigue showed moderate correlations with the MSQol-54 energy and emotional wellbeing scales and the RNLI. Overall impact of MS showed moderate correlations with the RNLI, the LMSQoL, MSQol-54 social function, and overall quality of life. The factor scores were all highly related to each other and were more highly related to each other than to the single point instruments. Fatigue and mobility were more highly related to each other than to overall impact of MS.

    Table 4

    Spearman’s rank correlations between median MSSID factor scores for weeks 9–12 and single point instruments collected at week 12

    Responsiveness

    Effect sizes for those who got better were moderate for MSSID scores and small for the MSQol-54 and LMSQol scores with one exception (emotional role limitations) (table 5). Effect sizes for those who got worse were small for the MSSID, MSQoL-54, and LMSQoL scores. The effect sizes were in the expected direction for MSSID and LMSQoL scores, but did not show a consistent pattern for MSQoL-54 scores.

    Table 5

    Effect sizes for median MSSID scores, MSQol-54, and LMSQoL scores for those who got better or worse in the first 4 weeks of the study

    DISCUSSION

    This study has provided initial support for the reliability, validity, and responsiveness of the (MSSID). Response rates were similar but the rates of missing items were higher compared with other studies using the MSQol-54, the LMSQoL, and the MSIS-29.8,9,27 However, response rates and rates of missing data were similar to other diary studies,18,22 providing initial evidence for the diary’s feasibility and acceptability. None of the items had ceiling effects and only three items had floor effects. These three items, "weakness or heaviness in your legs", "problems with your bladder and bowel", and "feeling frustrated" were retained because the developmental work indicated that they are both common and important problems for people with MS. Further work will evaluate whether excluding these items has an impact on the psychometric properties of the MSSID.

    The majority of items in the MSSID focus on the physical symptoms of MS and their impact on daily activities, which is similar to the item content of the MSIS-29.9 However, like the MSIS-29, it also includes important items that address the emotional impact of the condition. The MSSID consists of three factors that measure mobility, the cognitive and emotional aspects of fatigue, and the overall impact of MS. These factors are internally reliable and their test–retest reliability was adequate for group comparisons for weekly mobility, fatigue, and overall impact scores, and individual comparisons for daily mobility and fatigue scores. The MSSID was able to discriminate between people with progressive and relapsing remitting MS, between those who used a walking aid and those who did not, and between those who had recently experienced a relapse and those who had not. It demonstrated expected moderate associations with appropriate scales on the MSQoL-54, the LMSQoL, the Barthel Index, the LHS, and the RNLI. Although there was some overlap in the constructs measured by the factor scores, they assess the impact of mobility, fatigue, and MS as a whole on daily activities.

    Compared with the MSQoL-54 and the LMSQoL, the responsiveness of the median weekly MSSID scores was similar or better. The MSSID was better able to detect short term improvements in functioning than worsening, which may be attributable to the floor effects in the three items. It was also more responsive to short term improvements in functioning than the MSQoL-54 and the LMSQoL. This is in contrast to a previous study that found an asthma diary to have lower responsiveness compared with an asthma questionnaire. The better short term responsiveness of the MSSID may be because it was explicitly designed to detect short term changes in functioning, whereas the MSQoL-54 and the LMSQoL were not. The MSSID thus provides a valuable complement to currently existing instruments in its measurement of short term changes and day to day fluctuations in functioning.

    The sample size used in the study was small compared with those used to develop the LMSQoL, the MSIS-29, and the MSQoL-54.8,9,26 However, it was expected to be sufficient to provide accurate estimates of the psychometric properties of the MSSID because sample sizes as low as 20 for reliability estimates and 40 for validity estimates did not alter the psychometric interpretation of the MSIS-29 compared with a sample size of 713.38 The sample was recruited from outpatient clinics, and a third of respondents experienced a relapse during the study, reflecting previous research that people with MS attending outpatient clinics are more likely to experience relapses or have greater disability for their age than community based populations.39 Therefore, further testing of the psychometric properties of the MSSID in a community based sample of people with MS would be valuable.

    This study suggests that the MSSID could be used in clinical trials of both disease modifying drugs and interventions to manage MS symptoms. The MSSID may be a useful tool to monitor the occurrence of relapses and measure their impact in a psychometrically robust way. The MSSID is likely to require more resources to collect and manage the data and further research is required to assess the feasibility of its use in the context of a randomised controlled trial. However, as symptom diaries are sometimes used in current trials of MS therapies15 but are rarely psychometrically evaluated, the MSSID represents a significant advance in this respect.

    The MSSID may also be useful in routine clinical practice in providing the patient with a prompt to inform clinicians about new symptoms or a worsening of existing symptoms, thus reducing the uncertainty around the occurrence of a relapse.16 It can also provide a graphical display of changes in the impact of symptoms over time and may be useful to clinicians in monitoring responses to treatment. Further work is needed to explore the most appropriate and feasible method of presenting MSSID data to clinicians and to examine clinicians’ and patients’ views of the utility of this data.

    CONCLUSIONS

    This study provided initial evidence that the MSSID is reliable, valid and responsive to change. It may provide a useful complement to currently available instruments to measure the outcomes of MS within clinical trials or clinical practice. Further research is needed to explore its feasibility and utility to clinicians in randomised controlled clinical trials and within routine clinical practice.

    Acknowledgments

    The authors would like to thank A Jacoby for permission to modify the diary from the cost utility study and her comments on the study design. We would also like to thank C Young, M Boggild, and M Johnson for their comments on the study design and for access to and recruitment of people with MS to the study. D Greenwood and S Walters provided statistical advice. Thanks also go to all the people who gave their time to participate in the research. J Greenhalgh was an MRC Training Fellow in Health Services Research 1997–2001.

    REFERENCES

    View Abstract
    • Web-only Appendix

      The questionnaire is available as a downloadable PDF (printer friendly file).

      If you do not have Adobe Reader installed on your computer,
      you can download this free-of-charge, please Click here

       

      Files in this Data Supplement:

      • [View PDF] - MS symptom and impact diary:
        Confidential symptom and impact diary for people with multiple sclerosis: about these questions
    • The appendix is available as a downloadable PDF (printer friendly file).

      If you do not have Adobe Reader installed on your computer,
      you can download this free-of-charge, please Click here

       

      Files in this Data Supplement:

    Footnotes

    • Competing interests: none declared

    Request permissions

    If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.