Statistics from Altmetric.com
Plohmann et al 1investigated the effects of computer training of attentional deficits in patients with multiple sclerosis. They conclude that “significant improvements of performance could almost exclusively be achieved by the specific training programmes”. The validity of this conclusion is called into question by severe methodical shortcomings of their study.
Before training, three baseline measurements of attentional functions were administered with 3 week intervals. For evaluation of training effects the median value of the three baseline measurements was compared with the values obtained after training. This statistical approach manifests a curious misunderstanding of the purpose of repeated baseline measurements. They serve to determine a baseline, that is, a rate of change occurring without any therapy. The critical value is therefore not the mean (or median) of the baseline measures but the difference between them. If therapy is efficient, the difference between pretherapy and post-therapy measurement should be greater than that between two consecutive baseline measurements. This crucial comparison is not presented.
The selection of the median of the three baseline measurements as starting point for calculation of improvements during the first training period poses further problems. If there was any improvement from the first to the third baseline measurement, the median is lower than the third measurement after which training began. This difference inflates apparent improvement in the first training period. It may feign specific training effects if the training group had a steeper baseline than the control group. A possible reason for different baselines are different severities of initial impairment. We (Motz, Grömminger, Göttert, Goldenberg, unpublished data) have administered the PASAT,2 another test of attentional capacities, four times with weekly intervals to 30 patients with chronic brain damage from different aetiologies. During intervals these patients did not receive any training of attentional functions. Thus, the repeated measurements determined a baseline without therapy. None the less, perfomance on PASAT improved from test to test. There was a negative correlation between intitial performance and improvement. Patients with poor initial performance improved more than those with better performance.
The allocation of patients to treatment groups in the study of Plohmannet al was not randomised. Patients were trained in those two functions that were affected most, and group comparisons were made between patients who had been trained in a function and those who were not. Thus the training group tended to start from a lower level of performance than the control group. Figures 2 and 3 of their paper illustrate this effect impressively. If, as suggested by our results with the PASAT, initial level of impairment has a systematic influence on improvement independently of any therapy, the allegedly specific training effect may be accounted for by the differences in initial level.
Whether or not the results of Plohmann’s et al study are reliable has clinical and ethical implications. Multiple sclerosis is one of the most common neurological diseases, and I have the suspicion that no other neurological disease has given rise to a comparable number of scientifically unfounded therapies and advice. The above critique raises the possiblity that computer assissted retraining of attention is one of them. It may be relatively harmless in that it has no organic side effects. None the less, if its efficacy cannot be proved, it would be a waste of money, time, and patients’ hopes.
Plohmann and Kappos reply:
We thank Goldenberg for his interest in our paper1-1. After having been actively involved in planning and conducting controlled trials in multiple sclerosis in the past 15 years, we can only agree that the risk of drawing wrong conclusions from unreliable data cannot be overstressed in this area. All the same we cannot follow Goldenberg’s reasoning. His critique is probably based on his own unpublished observations but is neither supported by the available literature nor by our own data.
His main critique is that the effect described in our paper may only reflect non-specific practice effects. Retest effects are dependent on the interval between test presentations and the population studied, and differ from test to test according to their respective stability and reliability. In our study we assessed patients with multiple sclerosis in a stable or eventually slightly progressive phase of their disease. For cognitively impaired patients with multiple sclerosis it has been shown in longitudinal studies that they lack practice effects compared to cognitively intact patients.1-2 1-3 In our data a possible but in no way significant practice effect was found between the first (T1) and second (T2) baseline (pretreatment) measurement and—with two exceptions (“divided attention” and “flexibility”)—not at all between T2 and T3. Clearly significant improvement of performance could only be seen between all three baseline measurements and T4 (Friedman two way analysis of variance, Wilcoxon signed ranks test). Also the assumption that the median might be lower than the third measurement and therefore might not be an adequate starting point for statistical comparisons is not supported by our data. In most of the control tests the median is equivalent to T3 or even larger.
Goldenberg’s second point is that the initial level of performance has a systematic influence on improvement independently of any therapy. Furthermore, on the basis of his results concerning the PASAT, he assumes a negative correlation between initial performance and improvement. As our patients had received specific training in those two functions that were severely impaired and thus started from a lower level of performance, Goldenberg claims that the specific training effects may be accounted for by the differences in initial level of impairment. For this reason we reanalysed our data. For each training group we built subgroups of patients according to their initial capability (low, medium, high) and printed the course of performance during the baseline measurement (T1-T3). Our results do not confirm Goldenberg’s assumption that initial level of performance has a systematic or inverse influence on improvement.
Nevertheless, as already stated in our original paper, due to the small sample size of our study the results do not allow adefinite assessment of the training and its effects in patients with multiple sclerosis; but they provide enough evidence to encourage further studies in a larger population of patients. Thus, we have started a prospective multicentre study in comparison with three Swiss neurological rehabilitation centres to further assess and possibly improve this kind computer assisted retraining of attentional functions.
The unpublished additional data mentioned in our letter have been submitted with this letter and are available on request (tables 1a-1c, figures 1 and 2).
If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.