Jump to comment:

• Meta-analysis by Xu et al. suffers from critical errors
  Evangelos Evangelou
  Published on: 13 April 2016
• Comments on "Meta-analysis of modifiable risk factors for Alzheimer's disease"
  Yu-Tzu Wu
  Published on: 13 April 2016

• Published on: 13 April 2016

  Meta-analysis by Xu et al. suffers from critical errors

  • Evangelos Evangelou, Assistant Professor
  • Other Contributors:
    • Lazaros Belbasis, Vanesa Bellou

  The meta-analysis by Xu et al is a valiant effort to map the evidence for modifiable risk factors of Alzheimer's disease (AD) (1). We acknowledge this huge effort, however, we have serious concerns regarding the systematic appraisal and the synthesis of the available data.

  On top of the critique by Drs. Wu and Brayne (e-letter), a comprehensive assessment of the article and its results can reveal critical error...

  Show More

  The meta-analysis by Xu et al is a valiant effort to map the evidence for modifiable risk factors of Alzheimer's disease (AD) (1). We acknowledge this huge effort, however, we have serious concerns regarding the systematic appraisal and the synthesis of the available data.

  On top of the critique by Drs. Wu and Brayne (e-letter), a comprehensive assessment of the article and its results can reveal critical errors in data collection and the analysis of the available evidence. An experienced methodologist could easily spot from the forest plots in the supplementary material that all risk factors are in the same direction, an observation that cannot be explained by chance.

  Therefore, we tried to re-analyze the meta-analyses using data reported in the supplementary material of this paper and, indeed, the summary effect estimates could not be replicated. We figured out that the authors have inadvertently used the extracted ORs (at least for binary exposures) derived from the individual studies as is, without transforming them into the logarithmic scale as it is required, leading to miscalculations of the summary effect sizes. Thus, most if not all inferences in the paper are based on these false summary estimates resulting in misleading conclusions.

  Besides the aforementioned analytical errors, we are afraid that the extraction of relevant data points from the primary studies may suffer from crucial errors as well. We summarise here potential problems for the association between "ever vs never alcohol use" and AD as it illustrated in supplementary material (page 184). Unfortunately, in 4 out of the 11 eligible studies the extracted estimates were wrong. Specifically, in 2 studies the authors included an estimate for the comparison of "wine drinkers vs no drinkers" even though an estimate for "ever vs never alcohol consumption" was available (2,3). In one study they inadvertently reported the estimate for a comparison of tuberculosis history (4) instead of the alcohol use, whereas in another study they extracted an estimate for an occupational exposure to alcohols and phenols (as organic solvents) (5). Moreover, it seems that the study "Lindsay, 2002" (3) has overlapping populations with the study "CHSA, 1994" (6) and therefore it should not be considered twice. Additionally, in one of the included studies (7) the authors have captured an estimate that is most probably is a typo. In the primary study, an OR of 4.10 (95% CI 0.60-80.50) is reported. Our back calculations (based on the point estimate and the lower confidence interval) have shown that the upper CI most probable was 28.05 and this was not corrected by Xu et al, influencing the weight of this study in the meta-analysis.

  Taking all these discrepancies into account we tried to repeat the meta-analysis for the exposure to alcohol and AD. The summary OR (estimated by our team using correct analytical methods, correct effect estimates and after the exclusion of the study with overlapping sample) was 0.79 (95% CI: 0.63-0.98) for fixed effects and 0.84 (95% CI: 0.57- 1.24) for random effects models with I2=57%. In contrast the authors presented a summary OR of 0.63 (95% CI, 0.48-0.79) under the fixed-effects model with I2=11.3%. We have observed similar discrepancies for several other risk factors. Obviously using these effects to calculate the population attributable fractions led to the false conclusion that 66% of the Alzheimer's disease cases could be prevented. Indeed, a more conservative and thorough estimation of the population attributable risk for modifiable risk factors of AD indicated that known risk factors are responsible for only 28.2% (95% CI, 14.2% to 41.5%) of total AD cases worldwide (8).

  Furthermore, inclusion of studies with overlapping samples was also observed in meta-analyses of key risk factors. For example, in the meta- analysis of type 2 diabetes mellitus, the authors included three studies on Cardiovascular Health Cognition Study (Kuller et al 2003, Becker et al 2009 and Irie et al 2008) and another two studies on Baltimore Longitudinal Study of Aging (Moffat et al 2004, Dal et al 2005). However, only the study with the longest follow-up period should be included in the analysis for each of these cohort studies. The same caveat was observed in the meta-analysis for educational attainment.

  Additionally, for other risk factors (e.g socioeconomic status and educational attainment) the authors have not harmonized the exposure of interest, presenting separate meta-analyses for "high level versus low level" and for "low level versus high level" of exposure.

  Finally, we should express our concerns regarding the search strategy applied from the authors. For example in the case of coffee or caffeine drinking, a previous published meta-analysis in 2015 (9) includes more studies and yields non-significant effect estimates (5 studies in the paper by Kim et al and 4 studies in the paper by Xu et al). The authors report a significant random-effects OR of 0.54 (95% CI: 0.39 to 0.69) compared to 0.78 [95% CI: 0.78 to 1.22) by Kim et al which is clearly non- significant. Several similar examples exist throughout the manuscript (e.g. current vs. never statin use, NSAIDs use, ever vs. never smokers).

  Having all these caveats in mind the editorial team of the journal should thoroughly re-assess all the available evidence presented in this paper.

  References

  1. Xu W, Tan LL, Wang H-F, Jiang T, Tan M-S, Tan LL, et al. Meta- analysis of modifiable risk factors for Alzheimer's disease. J Neurol Neurosurg Psychiatry. 2015;86(12):1299-306.

  2. Tyas SL, Manfreda J, Strain LA, Montgomery PR. Risk factors for Alzheimer's disease: a population-based, longitudinal study in Manitoba, Canada. Int J Epidemiol. 2001 Jun;30(3):590-7.

  3. Lindsay J, Laurin D, Verreault R, H?bert R, Helliwell B, Hill GB, et al. Risk factors for Alzheimer's disease: a prospective analysis from the Canadian Study of Health and Aging. Am J Epidemiol. 2002;156(5):445- 53.

  4. Harmanci H, Emre M, Gurvit H, Bilgic B, Hanagasi H, Gurol E, et al. Risk factors for Alzheimer disease: a population-based case-control study in Istanbul, Turkey. Alzheimer Dis Assoc Disord. 2003;17(3):139-45.

  5. Kukull WA, Larson EB, Bowen JD, McCormick WC, Teri L, Pfanschmidt ML, et al. Solvent exposure as a risk factor for Alzheimer's disease: a case-control study. Am J Epidemiol. 1995;141(11):1059-71.

  6. The Canadian Study of Health and Aging: risk factors for Alzheimer's disease in Canada. Neurology. 1994;44(11):2073-80.

  7. Harwood DG, Barker WW, Loewenstein DA, Ownby RL, St George-Hyslop P, Mullan M, et al. A cross-ethnic analysis of risk factors for AD in white Hispanics and white non-Hispanics. Neurology. 1999;52(3):551-6.

  8. Norton S, Matthews FE, Barnes DE, Yaffe K, Brayne C. Potential for primary prevention of Alzheimer's disease: An analysis of population- based data. Lancet Neurol. 2014;13(8):788-94.

  9. Kim Y-S, Kwak SM, Myung S-K. Caffeine intake from coffee or tea and cognitive disorders: a meta-analysis of observational studies. Neuroepidemiology. 2015;44(1):51-63.

  Conflict of Interest:

  None declared

  Show Less

  Conflict of Interest:
  None declared.

  • Back to top
• Published on: 13 April 2016

  Comments on "Meta-analysis of modifiable risk factors for Alzheimer's disease"

  • Yu-Tzu Wu
  • Other Contributors:
    • Carol Brayne

  The meta-analysis by Xu et al., (2015)[1] reviews over 300 papers and identified 93 modifiable risk factors for Alzheimer's disease. Among these risk factors, population attributable risks (PAR) of nine important factors were estimated to contribute to up to 66% of Alzheimer's disease cases globally. We acknowledge the great effort of this massive review but there are some substantial analytical issues which should be highl...

  Show More

  The meta-analysis by Xu et al., (2015)[1] reviews over 300 papers and identified 93 modifiable risk factors for Alzheimer's disease. Among these risk factors, population attributable risks (PAR) of nine important factors were estimated to contribute to up to 66% of Alzheimer's disease cases globally. We acknowledge the great effort of this massive review but there are some substantial analytical issues which should be highlighted.

  (1) Mixture of odds ratio (OR) and relative risk (RR) in the review Although the authors clearly knew the difference between odds ratio (OR) and relative risk (RR) they still combined these results and generated pooled estimates in the meta-analysis. It is generally considered problematic to combine these two epidemiological measures.[2] Odds ratios tend to provide a stronger effect size than relative risks with the two measures only being very close if the prevalence of a disease/disorder is very low. Dementia and Alzheimer's disease are not rare therefore this approach may have led to overestimation of the effect size of risk factors. Although the authors stratified cohort and case-control studies in the meta-analyses, they still reported overall pooled estimates. Given the large number of papers, it would have been possible to focusing only on studies reporting relative risk.

  (2) Grades of evidence Grades of evidence were based on heterogeneity and pooled population size. Although heterogeneity indicates consistency of the results across individual studies, a non-significant I-squared should not be taken as evidence of no heterogeneity.[3] In particular, observational studies generally depend on their research settings and use a variety of methods with diverse study populations. Apparently, similar results can mask considerable underlying variation.

  In the appendix, the authors further excluded some studies according to concerns about study quality and methods. This is an important methodological point and exclusion of studies requires a systematic approach in order for the reader to understand the decisions reviewers have made. Assessments of study quality should be included in "grades of evidence". This corresponds to the first point that the authors could have focused on studies using prospective designs and proper methodologies to provide robust estimates.

  (3) PAR estimation without taking inter-relationship of risk factors into account The PAR calculation did not take into account the inter-relationship between the nine factors leading to spuriously high PAR estimates (66%). Other studies[4,5] have addressed this recognised limitation and an updated analysis of the paper cited by Xu et al[6] was published precisely to correct this important concern. This reduces PAR to around 30%. This is because risk factors such as obesity/low BMI index, frailty/low BMI index (and indeed education and depression) are known to be highly related. Epidemiologists are well aware of the joke that with such naive analyses we claim to be able to prevent over 100% of disorders.

  Finally, some confusing factors such as "current smoking in Asian population" and "diabetes mellitus-II in Asian population" were included in the analysis. The authors suggest the PAR estimate was applied to global Alzheimer's disease cases, but these factors only focus on Asian populations. It is unclear whether it is appropriate to apply these globally.

  We suggest that the findings of this paper over-estimate the effect size and contribution of risk factors for dementia and Alzheimer's disease and that, unfortunately, this contributes to further confusion in wildly over-interpreted media headlines.

  References

  [1] Xu W, Tan L, Wang H-F, Jiang T, Tan M-S, Tan L, et al. Meta- analysis of modifiable risk factors for Alzheimer's disease. Journal of Neurology, Neurosurgery & Psychiatry. 2015. DOI: 10.1136/jnnp-2015- 310548

  [2] Bonita R, Beaglehole R & Kjellstrom T. WHO Basic Epidemiology. 2006. Available: apps.who.int/iris/bitstream/10665/43541/1/9241547073_eng.pdf

  [3] Higgins J & Green S. Cochrane Handbook for Systematic Reviews of Interventions. 2011. Available: handbook.cochrane.org/chapter_9/9_5_2_identifying_and_measuring_heterogeneity.htm

  [4] Norton S, Matthews FE, Barnes DE, Yaffe K, Brayne C. Potential for primary prevention of Alzheimer's disease: an analysis of population- based data. The Lancet Neurology. 2014;13(8):788-94.

  [5] Di Marco LY, Marzo A, Mu?oz-Ruiz M, Ikram MA, Kivipelto M, Ruefenacht D, et al. Modifiable lifestyle factors in dementia: a systematic review of longitudinal observational cohort studies. Journal of Alzheimer's Disease. 2014;42(1):119-35.

  [6] Barnes DE, Yaffe K. The projected effect of risk factor reduction on Alzheimer's disease prevalence. The Lancet Neurology. 2011;10(9):819- 28.

  Conflict of Interest:

  None declared

  Show Less

  Conflict of Interest:
  None declared.

  • Back to top