Objective To assess the prognostic value of practice effect on Paced Auditory Serial Addition Test (PASAT) in multiple sclerosis.
Methods We compared screening (day −14) and baseline (day 0) PASAT scores of 1009 patients from the FTY720 Research Evaluating Effects of Daily Oral therapy in Multiple Sclerosis (FREEDOMS) trial. We grouped patients into high and low learners if their PASAT score change was above or below the median change in their screening PASAT quartile group. We used Wilcoxon test to compare baseline disease characteristics between high and low learners, and multiple regression models to assess the respective impact of learning ability, baseline normalised brain volume and treatment on brain volume loss and 6-month confirmed disability progression over 2 years.
Results The mean PASAT score at screening was 45.38, increasing on average by 3.18 from day −14 to day 0. High learners were younger (p=0.003), had lower Expanded Disability Status Scale score (p=0.031), higher brain volume (p<0.001) and lower T2 lesion volume (p=0.009) at baseline. Learning status was not significantly associated with disability progression (HR=0.953, p=0.779), when adjusting for baseline normalised brain volume, screening PASAT score and treatment arm. However, the effect of fingolimod on disability progression was more pronounced in high learners (HR=0.396, p<0.001) than in low learners (HR=0.798, p=0.351; p for interaction=0.05). Brain volume loss at month 24 tended to be higher in low learners (0.17%, p=0.058), after adjusting for the same covariates.
Conclusions Short-term practice effects on PASAT are related to brain volume, disease severity and age and have clinically meaningful prognostic implications. High learners benefited more from fingolimod treatment.
- learning effect
- disability progression
- multiple sclerosis
This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0
Statistics from Altmetric.com
If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.
Multiple sclerosis (MS) is a chronic and debilitating disease affecting mainly adults early in their life. It is a progressive disease characterised by the formation of lesions in the brain and spinal cord. Symptoms of the disease are widespread and largely dependent on the location of the lesions and the extent of inflammatory and degenerative pathology within the central nervous system. The majority of patients manifest with a relapsing-remitting disease course but progress to a more progressive and degenerative stage of the disease within a decade.1 In addition to motor symptoms, cognitive impairment is prevalent in all stages of MS,2 3 including early relapsing-remitting disease and clinically isolated syndromes.4–7 The cognitive functions preferentially affected in MS include attention, information processing efficiency and speed, executive functioning, and long-term memory.3 Over the last two decades, the Paced Auditory Serial Addition Test (PASAT), as part of the Multiple Sclerosis Functional Composite score, has been the most commonly used test to evaluate attentional function in MS clinical trials.8–10 However, at repeated administrations, a significant practice effect has been reported resulting in increased mean PASAT scores in both the active and control groups over time.11 12 This practice effect makes interpretation of true treatment effects difficult10 13 and accentuates the ceiling effect in patients with higher baseline scores who anyway have limited room for further improvement in contrast to those with low baseline scores who can further improve at subsequent visits.13 Nevertheless, PASAT scores correlate with brain volume (BV) cross-sectionally.14 Brain volume loss (BVL) is increasingly recognised as a predictor of disability progression,15–17 but interpretation of the longitudinal association to PASAT scores is complicated by the observed practice effects. Recent studies have suggested that higher BV and cognitive reserve can be buffers to accelerated disability progression.18 19 We investigated the relation of the PASAT practice effect as a measure of learning ability and cognitive reserve with other disease characteristics, and determined whether this practice effect—as a dynamic measure—can inform prognosis of patients with MS.
In patients who participated in the FTY720 Research Evaluating Effects of Daily Oral therapy in Multiple Sclerosis (FREEDOMS) trial comparing fingolimod and placebo, we (1) assessed and quantified learning ability as expressed by change in PASAT score after short-term repeated administration accounting for baseline cognitive performance; (2) investigated the correlation of practice effect with baseline MS disease characteristics; and (3) determined whether the practice effect can predict on-study BVL, future disability worsening and treatment effect.
Study design and patients
This is a post-hoc analysis of data from the FREEDOMS20 study (n=1009, complete data set), a phase III, randomised, placebo-controlled study of fingolimod in patients with relapsing-remitting multiple sclerosis (RRMS). The study design and inclusion/exclusion criteria of the FREEDOMS study have been previously described.20 In brief, the study included patients diagnosed with RRMS (according to the revised McDonald criteria21) aged 18–55 years who had a score of 0–5.5 on the Expanded Disability Status Scale (EDSS), and one or more documented relapses in the past year, or two or more relapses in the past 2 years prior to randomisation. Patients were to be neurologically stable with no evidence of relapse or corticosteroid treatment within 30 days prior to randomisation. Eligible patients were randomised (1:1:1) to receive either fingolimod 0.5 or 1.25 mg once daily or placebo for 2 years.20 In all the analyses the two treatment arms are pooled together.
Standardised neurological assessments, including determination of EDSS score, were performed at baseline and every 3 months. MRI scans were obtained at baseline and months 6, 12 and 24 or at the end of study in patients prematurely discontinuing the study. MRI lesion activity and BVL were assessed at a central reading site (Medical Image Analysis Center, Basel, Switzerland) that remained blinded for clinical data and randomisation. Normalised brain volume (NBV) was assessed at baseline using ‘Structural Image Evaluation, using Normalization, of Atrophy, Cross-sectional’ (SIENAX). PASAT was administered at screening (day −14; two trials) and at baseline (day 0; one test). Per cent brain volume change (PBVC) between baseline and month 24 was determined using ‘Structural Image Evaluation, using Normalization, of Atrophy’ (SIENA). Disability worsening was defined as a ≥1-point increase in the EDSS score sustained for ≥6 months (1.5 points if baseline EDSS=0).
Patients were divided into four groups by quartiles of the PASAT score at screening (day −14; average of two trials), and the median value of PASAT change for each quartile was calculated. In each of these four groups, patients were further defined as ‘high learners’ if they had a PASAT score improvement above their group’s median, or ‘low learners’ if they had a PASAT score change below the median of their quartile group. To mitigate ceiling effects patients with screening PASAT scores of 60 (maximum value) and no change at baseline were counted as ‘high learners’. By defining learning ability within the screening quartile groups, we could take into account the fact that patients with higher screening scores have less room for further improvement (ceiling effect) compared with those who started with a low screening score.
The Wilcoxon rank-sum test was used to compare different baseline characteristics (age, clinical disease burden (EDSS), T2 lesion volume (T2LV) and NBV) between high and low learners. A multivariate logistic regression model with a forward selection was run to select factors independently associated with learning ability. Pearson’s correlation (r) was used to measure the correlations between PASAT score at screening and change in PASAT score with baseline age, EDSS, disease duration, T2 lesion load and NBV. A Cox proportional hazard model was used to assess if there was a difference in the risk of 6-month confirmed disability progression (CDP) over 2 years between high and low learners after adjusting for fingolimod treatment, NBV and screening PASAT score. The Cox model was used to assess if treatment effect differed based on the high versus low learner status, by including a treatment-by-learning status interaction in the model. Finally, a linear regression model was used to determine whether learning ability between screening and baseline is predictive of the rate of PBVC over 2 years, after adjusting for fingolimod treatment, NBV and screening PASAT score.
PASAT practice effects
Overall, a mean improvement of 3.18 (SD=6.44) was noted in the PASAT score from screening (45.38 (11.24), day −14) to baseline (48.56 (11.07), day 0), compatible with the well-known practice effect (figure 1A). The median change in PASAT score from screening to baseline was 5.0 for Q1, 3.5 for Q2 and Q3, and 0.5 for Q4 (figure 1B). A total of 513 patients were included in the ‘high-learners’ group and 496 in the ‘low-learners’ group (figure 1B).
Correlations of baseline findings and practice effect
The baseline characteristics of the total analysis cohort by learning ability are reported in table 1.
The PASAT score at screening was positively correlated with baseline PASAT score (r=0.83, p<0.0001) and NBV (r=0.26, p<0.0001), and was inversely correlated with PASAT change/practice effect (r=−0.31, p<0.0001; figure 1C). No correlation was found between NBV and unadjusted PASAT change (r=0.02, p=0.48).
High learners were younger than low learners (p=0.003) and had higher NBV (p<0.001; table 1). Moreover, high learners had lower clinical (EDSS score, p=0.03) and MRI (T2LV, p=0.009) disease burden at baseline (table 1). The multivariate analysis showed that NBV has the strongest association with learning ability: the OR=1.35 (p<0.001) indicated a 35% increase in the probability to be a high learner for every increase of NBV by 100 mm3.
Disease outcomes and treatment effect by practice effect
The PASAT score at screening, treatment and NBV were all associated with the probability of 6-month CDP in the univariate analysis, but when included in a multivariate model only treatment with fingolimod (HR 0.56, p<0.001 vs placebo) and NBV (HR 0.76, p=0.006) were independently associated with a decreased risk of 6-month CDP (table 2). The risk of 6-month CDP was not significantly affected by the learning ability. However, when a model with a treatment-by-learner status interaction was applied, the interaction term was significant (p=0.05), indicating that the fingolimod treatment effect is stronger in high learners (60% risk reduction, HR=0.40, p<0.001) than in low learners (21% risk reduction, HR=0.79, p=0.35; figure 2).
PBVC over 2 years was lower in patients treated with fingolimod (PBVC=−0.87%) than those treated with placebo (PBVC=−1.28%; difference −0.39%, p<0.001; table 3). It was also lower in patients with higher NBV values at baseline (p=0.006) and higher screening PASAT scores (p=0.038) (table 3). Patients from the high-learner group experienced less PBVL than those from the low-learner group (p=0.01, table 3). Treatment, NBV, screening PASAT score and learning ability all correlated with PBVL in both the univariate and multivariate analyses. Being in the high PASAT learner group was therefore independently associated with a lower PBVL throughout the study period, and even after adjusting for screening PASAT score and baseline NBV the difference (−0.17%) remained borderline significant (p=0.058; figure 3).
The practice effect seen with PASAT is usually considered a limitation of this test, because it can potentially confound the interpretation of a treatment effect on cognitive performance in clinical trials.22 Because this effect is most pronounced in the first repetitions of the test, most studies require to repeat the test at least two to three times before treatment start. However, the improvement in performance over time observed at repeated administrations is reflective of active learning.13 23 The driving hypothesis of our study was that this learning effect induced by repeated testing, besides being an interfering variable, would by itself be a valuable measure of cognitive performance. To date the potential clinical relevance and implications of this learning effect and the association of practice effect and learning ability with disease outcomes have been poorly investigated. A study in patients with clinically isolated syndrome on disease-modifying therapy assessed the temporal association between MRI measures and PASAT performance and reported higher PASAT learning effects in patients with lower rates of grey matter atrophy.24
Based on a large, randomised controlled study, we confirm the existence of a significant practice effect on PASAT scores13; patients’ cognitive performance improved on average when the test was repeated 14 days after the first PASAT trial. Patients with high PASAT scores at screening exhibited lower PASAT changes compared with those with a low screening PASAT score. This inverse correlation indicates the need to adjust the PASAT change for screening PASAT scores in order to correctly identify the patients who show improvements at baseline. Assessing the change in PASAT score, as compared with the median screening PASAT score within a quartile, allowed to quantify the learning ability as a measure of cognitive reserve adjusting for the impact of baseline performance.
Patient baseline demographic and disease characteristics are also associated with cognitive performance as measured by PASAT. High PASAT scores at baseline corresponded to younger patients who also had high NBV and lower disease burden at baseline. Thus, the learning ability of patients with RRMS, as measured using the practice effect on PASAT, is reflective of age and MS disease severity (EDSS score, BV and T2LV) at baseline. It is known that performance in PASAT is influenced by intelligence and mathematical ability,13 25 which could contribute to the ceiling effect. Also, comorbidities such as depression and/or anxiety13 and occurrence of a recent MS relapse may affect the cognitive status of the patient. However, it is difficult to measure the impact of these variables, and therefore adjustment for these factors is difficult to achieve. The FREEDOMS trial did not include patients with pre-existing serious psychiatric conditions. Although assessment of neurological disability (changes in functional system and EDSS score) was performed to confirm a relapse, neurocognitive status was not evaluated at the time or immediately after the relapse. Deficits in information processing speed and working memory is the most prominent aspect of cognitive impairment in patients with MS. The association of cognitive status and disability progression as assessed by change in EDSS score is not clearly reported in the literature, and is further complicated by the fact that there is lack of a standard tool to measure cognitive impairment. It has been reported that patients with cognitive impairment at baseline had worsening EDSS score26 27 and were at significantly higher risk of reaching EDSS >4 as compared with cognitively preserved patients.28 However, some studies suggest that cognitive decline is better correlated with brain atrophy measures, particularly deep grey matter atrophy, than EDSS progression.29 30
It has been previously reported that treatment effect on brain atrophy correlates with the effect on disability progression over 2 years and predicts future disability progression independent of the effect on active MRI lesions in patients with RRMS.31 Furthermore, a correlation between treatment effect of fingolimod on BVL and its corresponding effect on future disability progression has also been reported.32 In patients treated with fingolimod, 2-year disability progression was associated with the rate of BVL over 2 years, and treatment effect of fingolimod on PBVC was predictive of future disability.32 We now report the association of fingolimod treatment effect on disability progression based on the learning ability of patients. In both high and low learners, treatment with fingolimod reduced the risk of disability progression, with stronger effects in high learners. After adjusting for the practice effect, high learners showed better prognosis in terms of both BVL and CDP over 2 years. After accounting for the treatment effect of fingolimod and baseline NBV, PASAT and learning ability had however no additional explanatory value for the prediction of disability progression in patients with RRMS. The lack of correlation between unadjusted PASAT change and NBV was due to the dependence of PASAT change on the screening value. This supports the importance and stronger relevance of NBV (also called as brain reserve) in predicting disability progression in patients with RRMS compared with PASAT. Our data contradict the findings of Raghupathi et al 33 that stipulate PASAT as an independent predictor of disability progression in an integrated analysis including data from the AFFIRM, DEFINE, CONFIRM and ADVANCE trials. Others have reported that higher brain reserve attenuates the impact of disease on cognitive efficiency, as measured by performance in PASAT.34 35 Our results, along with the findings from these previous reports, further suggest that both brain reserve and cognitive reserve protect against disease-related disability and cognitive decline. Interestingly, baseline PASAT score, learning ability and treatment were independent predictors of BVL.
In conclusion, our results suggest that greater cognitive reserve, as indicated by high baseline PASAT score, is a good prognostic indicator and that a greater ability to learn is associated with better treatment response and better disease outcomes in the future.
The authors thank Uma Kundu (Novartis Healthcare, Hyderabad, India) for providing medical writing assistance, which includes literature search, writing the manuscript's first draft and revision of subsequent drafts as per authors' comments, preparation of tables and figures as per journal guidelines, coordination of author comments, and preparation for submission of the article.
Contributors MPS contributed to the conception, design and execution of the study, statistical analysis, and drafting of the manuscript's outline and critical revision of subsequent drafts of the manuscript, and is responsible for the overall content of the manuscript. NdS contributed to the conception of the study, interpretation of data and critical revision of the manuscript. GG contributed to the conception of the study, analysis and interpretation of data, and critical revision of the manuscript. DL contributed to the interpretation of data and critical revision of the manuscript. DP-M contributed to the conception of the study, interpretation of data, drafting and critical revision of the manuscript. DAH contributed to the execution of the study, statistical analysis and interpretation of data, and critical revision of the manuscript. LK, as the principal investigator of the FREEDOMS trial, was responsible for supervising the trial. He also contributed to the conception, design and execution of the study, interpretation of data, and critical review of manuscript. DT contributed to the conception, design and execution of the study, interpretation of data, and critical revision of the manuscript.
Funding The authors have not declared a specific grant for this research from any funding agency in the public, commercial or not-for-profit sectors.
Competing interests This study was supported by Novartis Pharma AG, Basel, Switzerland. MPS received compensation for serving on the Scientific Advisory Boards of Teva, Genzyme, Novartis, Roche and Vertex; funding for travel or speaker honoraria from Merck Serono, Teva, Genzyme, Novartis, Biogen and Roche; consultancy from Merck Serono, Biogen, Teva, Genzyme, Roche, GeNeuro, MedDay and Novartis; Speakers' Bureaus from Teva, Merck Serono, Biogen, Novartis and Genzyme. NdS (University of Siena) has served on scientific advisory boards and steering committees of clinical trials for Merck Serono SA, Novartis Pharma AG and Teva, and has received support for congress participation or speaker honoraria from Bayer Schering AG, Biogen Idec, Merck Serono SA, Novartis Pharma AG, Sanofi-Aventis and Teva. GG has received compensation for serving as a consultant or speaker for or has received research support from AbbVie, Almirall, Atara Bio, Bayer Schering Healthcare, Biogen Idec, Canbex, Eisai, Elan, Five Prime Therapeutics, Genzyme, Genentech, GlaxoSmithKline, Ironwood Pharmaceuticals, Merck Serono, Novartis, Pfizer, Roche, Sanofi-Aventis, Synthon BV, Teva Pharmaceutical Industries, UCB and Vertex Pharmaceuticals. DL has participated on advisory boards/received consultancy/research grants or is in the Speaker Bureau for Bayer, Merck, Novartis, Teva, Excemed, Roche and Biogen. DP-M and DAH are employees of Novartis. LK has received no personal compensation. LK’s institution (University Hospital Basel) has received the following exclusively for research support: steering committee, advisory board and consultancy fees (Actelion, Addex, Bayer HealthCare, Biogen Idec, Biotica, Genzyme, Lilly, Merck, Mitsubishi, Novartis, Ono Pharma, Pfizer, Receptos, Sanofi, Santhera, Siemens, Teva, UCB and Xenoport); speaker fees (Bayer HealthCare, Biogen Idec, Merck, Novartis, Sanofi and Teva); support for educational activities (Bayer HealthCare, Biogen, CSL Behring, Genzyme, Merck, Novartis, Sanofi and Teva); royalties (Neurostatus Products); licence fees for Neurostatus Products; and grants (Bayer HealthCare, Biogen Idec, European Union, Merck, Novartis, Roche Research Foundation, Swiss MS Society and Swiss National Research Foundation). DT is an employee of Novartis.
Patient consent Not required.
Provenance and peer review Not commissioned; externally peer reviewed.
Data sharing statement This is a post-hoc analysis of data from patients who had participated in the phase III clinical trial (FREEDOMS). Any data not provided in the article, including statistical analyses and assumptions, may be shared at the request of other investigators.