Article Text

Download PDFPDF

How effective is effective enough?
Free
  1. Tomas Kalincik1,2,
  2. Amy Brodtmann3,4,5
  1. 1 CORe, Department of Medicine, University of Melbourne, Melbourne, Victoria, Australia
  2. 2 Neuroimmunology Centre, Department of Neurology, Royal Melbourne Hospital, Melbourne, Victoria, Australia
  3. 3 Department of Neurology, The Royal Melbourne Hospital, Parkville, Victoria, Australia
  4. 4 Central Clinical School, Monash University, Clayton, Victoria, Australia
  5. 5 Department of Medicine, University of Melbourne, Melbourne, Victoria, Australia
  1. Correspondence to Professor Tomas Kalincik, Neuroimmunology Centre, Royal Melbourne Hospital; 635 Elizabeth St, Melbourne, Victoria, Australia; tomas.kalincik{at}unimelb.edu.au

Statistics from Altmetric.com

Request Permissions

If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.

On a clinically meaningful treatment effect in Alzheimer’s disease

The paper by Goldberg et al explores the utility of reporting standardised effect size and per cent change in randomised clinical trials.1 They show that number-needed-to-treat or area under the receiver-operator characteristic curve—conventional measures of absolute difference between treatment groups—are reflected by the standardised effect size, but not the per cent change. Several Alzheimer’s disease trials of anti-amyloid monoclonal antibodies and lifestyle interventions have reported that interventions reduce cognitive decline, based on per cent changes and p values. However, these studies reported relatively small standardised effect sizes and considerable number-needed-to-treat.

A d of 0.15–0.19 can only be described as small, but it is also important to understand the measure from which this was derived. The Clinical Dementia Rating Sum of Boxes (CDR-SOB) is a standard metric for dementia trials: a cognitive and functional impairment scale scored via interview with participant and informant. Many triallists acknowledge its inherent flaws—ordinal nature and susceptibility to informant and rater bias.2 It is hard to find a comparative outcome measure as dependent on participant and carer report outside dementia.

JAMA Guide to Statistics and Methods provides valuable insights into reporting of effects in clinical research. An absolute difference, such as number-needed-to-treat, provides a clinically accessible measure of treatment effect. A relative difference helps the audience contextualise the absolute difference to the reference scenario (eg, placebo). Importantly, a 20% difference translates into a much smaller absolute effect in more cognitively impaired cohorts than in relatively preserved cohorts.

In testing scientific hypotheses, one evaluates the observed effects in relation to their variability. If this difference-to-variance ratio is sufficiently high, a test of statistical inference rejects the null hypothesis and the alternative is accepted. This is done with the help of a p value or, more appropriately, point and interval estimates of the effect size (eg, mean, 95% CI). The effect variability is embedded within the number-needed-to-treat and standardised effect size (eg, Cohen’s d), but not in the per cent difference.

The authors point out the important difference between a statistical proof of effect and a clinically meaningful effect. Defining a minimum clinically important difference is not trivial and includes perspectives of utility (impact on patients’ lives), clinical consensus and distribution of the studied metric.3 Indeed, a 1 to 2-point change in the CDR-SOB is regarded as a minimally clinically important difference, which is far greater than any of the reported differences in anti-amyloid monoclonal antibody studies to date.4 Donepezil, a symptomatic treatment licensed over 20 years ago, has comparable cognitive benefit at one-hundredth of the cost of these agents. This has prompted some researchers to call for a broader perspective in AD studies, where multiple pathologies are targeted.

Results of clinical studies can rarely be condensed into a single value. It is substantially more informative to consider differences between compared interventions from multiple angles. The ratio of a mean difference to its variability enables us to test scientific hypotheses statistically. Clinically, both absolute and relative effects help neurologists understand the importance of findings in the context of a minimum clinically meaningful difference and the natural course of Alzheimer’s disease.

Ethics statements

Patient consent for publication

Ethics approval

Not applicable.

References

Footnotes

  • Contributors TK and AB drafted and edited the text.

  • Funding The authors have not declared a specific grant for this research from any funding agency in the public, commercial or not-for-profit sectors.

  • Competing interests TK served on scientific advisory boards for MS International Federation and WHO, BMS, Roche, Janssen, Sanofi Genzyme, Novartis, Merck and Biogen, steering committee for Brain Atrophy Initiative by Sanofi Genzyme, received conference travel support and/or speaker honoraria from WebMD Global, Eisai, Novartis, Biogen, Roche, Sanofi-Genzyme, Teva, BioCSL and Merck and received research or educational event support from Biogen, Novartis, Genzyme, Roche, Celgene and Merck. AB serves on scientific advisory boards for Eisai (lecanemab), Biogen (aducanumab) and Roche (gantenerumab), as was a PI for the TRAILBLAZER (Lilly) and ENGAGE (Biogen) studies.

  • Provenance and peer review Commissioned; internally peer reviewed.

Linked Articles