Article Text

other Versions

PDF
Effect sizes can be misleading: is it time to change the way we measure change?
  1. Jeremy C Hobart1,2,
  2. Stefan J Cano1,2,
  3. Alan J Thompson2
  1. 1The Neurological Outcome Measures Unit, Peninsula College of Medicine and Dentistry, Devon, UK
  2. 2UCL Institute of Neurology, London UK
  1. Correspondence to Dr J C Hobart, Department of Clinical Neuroscience, Peninsula College of Medicine and Dentistry, Room N16 ITTC Building, Tamar Science Park, Davy Road, Plymouth, Devon PL6 8BX, UK; jeremy.hobart{at}pms.ac.uk

Abstract

Objectives Previous comparisons of the ability to detect change in the Barthel Index (BI) and Functional Independence Measure motor scale (FIMm) have implied these two scales are equally responsive when examined using traditional effect size statistics. Clinically, this is counterintuitive as the FIMm has greater potential to detect change than the BI and raises concerns about the validity of effect size statistics as indicators of rating scale responsiveness. To examine these concerns, in this study a sophisticated psychometric analysis was applied, Rasch measurement to BI and FIMm data.

Methods BI and FIMm data were examined from 976 people at a single neurorehabilitation unit. Rasch analysis was used to compare the responsiveness of the BI and FIMm at the group comparison level (effect sizes, relative efficiency, relative precision) and for each individual person in the sample by computing the significance of their change.

Results Group level analyses from both interval measurements and ordinal scores implied the BI and FIMm had equivalent responsiveness (BI and FIMm effect size ranges −0.82 to −1.12 and −0.77 to −1.05, respectively). However, individual person level analyses indicated that the FIMm detected significant improvement in almost twice as many people as the BI (50%, n=496 vs 31%, n=298), and recorded less people as unchanged on discharge (FIMm=4%, n=38; BI=12%, n=115). This difference was found to be statistically significant (χ2=273.81; p<0.000).

Conclusions These findings demonstrate that effect size calculations are limited and potentially misleading indicators of rating scale responsiveness at the group comparison level. Rasch analysis at the individual person level showed the superior responsiveness of the FIMm, supporting clinical expectation, and its added value as a method for examining and comparing rating scale responsiveness.

  • Multiple sclerosis
  • rehabilitation
  • scales

Statistics from Altmetric.com

Footnotes

  • Competing interests None.

  • Ethics approval This study was conducted with the approval of the National Hospital for Neurology and Neurosurgery.

  • Provenance and peer review Not commissioned; externally peer reviewed.

Request permissions

If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.

Linked Articles

  • Editorial commentary
    I S J Merkies R A C Hughes