On a clinically meaningful treatment effect in Alzheimer’s disease

The paper by Goldberg et al explores the utility of reporting standardised effect size and per cent change in randomised clinical trials.1 They show that number-needed-to-treat or area under the receiver-operator characteristic curve—conventional measures of absolute difference between treatment groups—are reflected by the standardised effect size, but not the per cent change. Several Alzheimer’s disease trials of anti-amyloid monoclonal antibodies and lifestyle interventions have reported that interventions reduce cognitive decline, based on per cent changes and p values. However, these studies reported relatively small standardised effect sizes and considerable number-needed-to-treat.

A d of 0.15–0.19 can only be described as small, but it is also important to understand the measure from which this was derived. The Clinical Dementia Rating Sum of Boxes (CDR-SOB) is a standard metric for dementia trials: a cognitive and functional impairment scale scored via interview with participant and informant. Many triallists acknowledge its inherent flaws—ordinal nature and susceptibility to informant and rater bias.2 It is hard to find a comparative outcome measure as dependent on participant and carer report outside dementia.

JAMA Guide to Statistics and Methods provides valuable insights into reporting of effects in clinical research. An absolute difference, such as number-needed-to-treat, provides a clinically accessible measure of treatment effect. A relative difference helps the audience contextualise the absolute difference to the reference scenario (eg, placebo). Importantly, a 20% difference translates into a much smaller absolute effect in more cognitively impaired cohorts than in relatively preserved cohorts.

In testing scientific hypotheses, one evaluates the observed effects in relation to their variability. If this difference-to-variance ratio is sufficiently high, a test of statistical inference rejects the null hypothesis and the alternative is accepted. This is done with the help of a p value or, more appropriately, point and interval estimates of the effect size (eg, mean, 95% CI). The effect variability is embedded within the number-needed-to-treat and standardised effect size (eg, Cohen’s d), but not in the per cent difference.

The authors point out the important difference between a statistical proof of effect and a clinically meaningful effect. Defining a minimum clinically important difference is not trivial and includes perspectives of utility (impact on patients’ lives), clinical consensus and distribution of the studied metric.3 Indeed, a 1 to 2-point change in the CDR-SOB is regarded as a minimally clinically important difference, which is far greater than any of the reported differences in anti-amyloid monoclonal antibody studies to date.4 Donepezil, a symptomatic treatment licensed over 20 years ago, has comparable cognitive benefit at one-hundredth of the cost of these agents. This has prompted some researchers to call for a broader perspective in AD studies, where multiple pathologies are targeted.

Results of clinical studies can rarely be condensed into a single value. It is substantially more informative to consider differences between compared interventions from multiple angles. The ratio of a mean difference to its variability enables us to test scientific hypotheses statistically. Clinically, both absolute and relative effects help neurologists understand the importance of findings in the context of a minimum clinically meaningful difference and the natural course of Alzheimer’s disease.