Article Text
PDF
Abstract
OBJECTIVE Necropsy confirmed clinical diagnostic accuracy for Alzheimer’s disease is claimed to exceed 90%. This figure contains two fallacies; it includes cases in which Alzheimer’s disease exists with other diseases affecting cognition and the studies that report these figures excluded cases without necropsy (verification bias). The effect of these errors is estimated.
METHODS Data were taken from the University of Western Ontario Dementia Study, a registry of dementia cases with clinical and psychometric follow up to necropsy based in a university memory disorders clinic with secondary and tertiary referrals. Data were available on 307 patients; 200 (65%) had clinically diagnosed Alzheimer’s disease, 12 (4%) vascular dementia, 47 (15%) mixed dementia, and 48 (16%) had other diagnoses. One hundred and ninety two of 307 cases (63%) died and 122 of 192 fatalities (64%) had necropsies. The pathological material was interpreted in two ways, allowing and disallowing coexistent disease in making a diagnosis of Alzheimer’s disease. In cases without necropsy, progressive cognitive loss was used as a marker for degenerative dementia. The outcome measures of interest were the positive predictive value of a clinical diagnosis of Alzheimer’s disease allowing and disallowing coexistent diseases and with and without correction for cases that were not necropsied.
RESULTS The clinical diagnoses differed significantly between the population who died and those who did not. In cases without necropsy, 22% had no dementia on follow up, concentrated in early cases and men, showing considerable scope for verification bias. The positive predictive value of a diagnosis of Alzheimer’s disease was 81% including coexistent diseases, falling to 44% when limited to pure cases. Combined, these factors reduce the positive predictive value to 38% for pure Alzheimer’s disease.
CONCLUSIONS Correction for dual pathology and verification bias halves the positive predictive value of the clinical diagnosis of Alzheimer’s disease. Data derived from necropsy studies cannot be extrapolated to the whole population. This has important implications including uncertainty about diagnosis and prognosis and a dilution effect in therapeutic trials in Alzheimer’s disease.
- Alzheimer’s disease
- necropsy
- diagnostic accuracy
- verification bias
Statistics from Altmetric.com
Studies on the accuracy of the clinical diagnosis of Alzheimer’s disease typically report positive predictive values between 71% and 88%.1-9 These figures may be flattering for two reasons.
Firstly, Alzheimer’s disease is fatal, although it is often not thought of in this way. None of the studies reporting diagnostic accuracy waited until their whole cohort had died. Thus the necropsy population will be biased towards those with fatal conditions, which will artefactually raise the apparent diagnostic accuracy. This is a form of verification bias,10-12 in which the decision to perform a gold standard diagnostic test depends on the result of the test under evaluation. The difference here is that the selection bias is introduced by the condition’s natural history rather than by the test under examination.
Secondly, most series regard cases in which the pathological changes of Alzheimer’s disease coexist with other conditions affecting cognition to be true positives.1-4 6-8 13 This is important because cases with two or more conditions affecting cognition may progress at different rates from cases with Alzheimer’s disease alone and may respond less well to drugs directed at Alzheimer’s disease, a matter with important implications for drug trials. This is a fallacy of dual diagnosis.
We have reanalysed the data from the University of Western Ontario Dementia Study to estimate the magnitude of these factors. We have recently shown that in histologically established cases of pure Alzheimer’s disease, cognition declines at 12.8% per year during the linear phase of decline as measured by the extended scale for dementia (ESD),14 a cognitive rating scale based on the Mattis dementia rating scale.15 Using this as a criterion for progressive dementia and the first percentile of the scores achieved by control subjects as a criterion for dementia that may not be progressive (for example, vascular dementia) we analysed the data to estimate the proportion of cases clinically diagnosed as having a dementing illness, but who prove to have no dementia and who rarely come to necropsy. In addition, we assessed the effect of modifying the necropsy criteria to count only pure Alzheimer’s disease as a true positive diagnosis.
Methods
The University of Western Ontario Dementia Study is a 17 year old prospective clinicopathological study of dementia based in the memory clinic of a secondary and tertiary teaching hospital. Cases were typically secondary referrals from patients’ family physicians and controls were derived from patients’ spouses or from normal volunteers. Only data from subjects seen by VH were included here and these numbered 163 controls and 307 cases. Alzheimer’s disease was diagnosed according to standard criteria.16 17 Clinical diagnoses made were grouped together as Alzheimer’s disease, vascular dementia, mixed dementia, and other diagnoses. The diagnoses used in this analysis are those made after the initial consultation and investigations as would typically be the case for the commencement of therapy, entry into drug trials, or discussion of the diagnosis and prognosis with the patient. The group of other diagnoses included cases not thought to have dementia, such as those with depression etc and those with less common causes of dementia. Further details of the study have been published previously.2
Confirmation of the diagnosis by necropsy was available in 122 (64%) of the 192 fatalities amongst the 307 cases. All cases and controls were followed up at six-monthly intervals using the ESD.18-20
Necropsy diagnoses were based on modern pathological criteria for Alzheimer’s disease, diffuse Lewy body disease, and Parkinson’s disease.3 21 22 These have evolved from earlier criteria for Alzheimer’s disease23 and all our pathological material was reviewed during 1994–5 to update the pathological diagnoses. The pathological diagnoses of vascular dementia and mixed dementia are more subjective as no widely accepted criteria exist. Cases of mixed dementia (mixed dementia refers throughout to vascular lesions coexisting with Alzheimer’s disease) met histopathological criteria for Alzheimer’s disease,3 21 but also had sufficient infarcts in the pathologists’ opinion to contribute to the dementia. Similarly, cases of vascular dementia had no other cause for the dementia but had sufficient infarcts, in the pathologists’ opinion, to cause dementia. As the smallest infarct volume that can affect cognition may be much lower, we analysed the necropsy findings in two ways. Firstly, we used the original pathologists’ diagnoses, which ignored small amounts of infarction in making a final diagnosis and in which cases with Alzheimer’s disease and one or more other relevant diagnoses were counted as true positive cases of Alzheimer’s disease. This scheme corresponds to the criteria we have previously reported,2 and to most of the reported literature. Next, we reclassified all cases with any volume of supratentorial infarction (other than microscopical infarcts seen in association with congophilic angiopathy in cases of Alzheimer’s disease, which we have treated as part of the Alzheimer’s disease) as mixed dementia if they also met criteria for Alzheimer’s disease and reclassified all the other cases with Alzheimer’s disease and one or more other diagnoses as “other dementia” so that the pathological category of Alzheimer’s disease was restricted to pure Alzheimer’s disease. As there is currently debate over the aetiology of hippocampal sclerosis these were classified as “other dementia” rather than vascular dementia. There were only five such cases and their precise classification did not substantially affect our findings. We then compared these results with the results obtained using the original classification to obtain an estimate of the effect of including and excluding dual diagnoses from a diagnosis of Alzheimer’s disease.
To extend these findings to all clinically diagnosed cases included in this report we first established whether the patient population that had necropsy was the same as the population that died without necropsy. If biologically similar, the diagnostic accuracy for the necropsy confirmed group could reasonably be extrapolated to those who died without necropsy. We compared these two groups for clinical diagnosis, year of onset, and year of death. Unexpectedly, the group of patients who had died without having a necropsy differed in clinical diagnoses from the necropsied group. Consequently this group was added to the patients who were still alive and these were treated as one group.
In those who are still alive, or who died without necropsy, the clinical diagnosis cannot be confirmed. However, the long follow up available in the University of Western Ontario Dementia Study allows cases to be divided into those with and without a progressive dementia. Similarly, the large amount of control data allows patients to be divided into those who were already demented when first seen and those whose dementia developed while under review. Using these simple divisions, we have estimated the proportion of cases in whom dementia was diagnosed at first assessment, but who never progressed and who therefore did not have a progressive dementia. We used these data to adjust the proportion of the whole population represented by the pathologically confirmed cases for those cases that did not have a progressive dementia.
Having established that within the population of those diagnosed at initial assesssment as having a progressive dementia, there were some patients who did not progress, we compared these patients with those who did progress to see if we could establish any reason for our failure to classify the non-progressors correctly.
Results
There were 163 controls and 307 cases in whom a diagnosis had been made by VH according to standard criteria. In the 307 cases, clinical diagnoses were made of Alzheimer’s disease in 200, vascular dementia in 12, mixed dementia in 47, and other diagnoses in 48. Of the cases, 192 have died and 122 (64%) of these had necropsy. Table 1shows the diagnoses in these groups. Data from the control subjects were used only for their ESD data. Of the controls, only 25 (15%) have died and of these three had necropsy. One showed ischaemic changes and the others were normal.
Cases entering the study, dying, and undergoing necropsy divided by clinical diagnosis
Follow up by ESD, excluding cases who had no ESD examinations, was for a median of 2.0 years, interquartile range 0.5–3.75 years, range 0–11.6 years. Those who died were followed up for a median of 1.75 years, range 0–9.6 years, and those who remained alive were followed up for a median of 2.6 years, range 0–11.6 years, p=0.0006 (Mann-Whitney U test). Those who had necropsies were followed up for a median of 1.7 years, range 0–9.6 years and those who died without necropsy were followed up for a median of 1.9 years, 0–6.5 years (p=0.17).
To establish how far the results from the necropsy group could be extrapolated, the clinical diagnoses in this group were first compared with those who died without necropsy. These differed significantly (χ2=15.6, p=0.0014). Those who had necropsy also differed from those still alive (χ2=17.1, p=0.0007). Similarly, the clinical diagnoses in those who died with necropsy examinations differed from those made in patients without necropsy, alive or dead (χ2=20.6, p=0.0001). There was no significant administrative difference, particular date of entry into the study, or date of death, that could explain these differences, suggesting that the pathological diagnoses made in those with necropsies cannot be extrapolated to those without necropsies, alive or dead, as there may be a systematic biological difference between these two groups.
To divide cases without necropsy into demented and not demented groups we used the first percentile (209) of the range of scores on the ESD by normal subjects (1783 ESD observations in 163 controls, median 244, range 141–250, first percentile 209). The University of Western Ontario Dementia Study has also recently reported the invariable steady deterioration seen in Alzheimer’s disease when patients are regularly reassessed using the ESD.14 Figure 1 shows this for all histologically established cases of pure Alzheimer’s disease. Some fluctuation and apparent improvement is evident in these data emphasising the importance of follow up. Criteria for dementia were therefore one of the following: a score over 100 and less than 209 at entry with subsequent evidence of progression consistent with histologically established Alzheimer’s disease14 ; a score under 100 at entry (progression was not specifically sought in these cases because of the floor effect); a score of 209 or more at entry only if there was subsequent progression below 209 in a manner consistent with Alzheimer’s disease.14
A plot of change in ESD score against time for all the serial observations in the cases of pure Alzheimer’s disease shows that all cases deteriorate steadily with time.
Of the 146 cases without necropsy in whom a diagnosis of Alzheimer’s disease, vascular dementia, or mixed dementia was made, seven had no ESD scores. Thus 139 could potentially be assessed this way. Figure 2shows the progression of the ESD against time in these cases. It shows a group of cases of dementia, identified by their initial clinical assessment and investigations, who did not progress. The ESD exhibits a considerable ceiling effect and a score in the normal range does not exclude early Alzheimer’s disease. Of the 139 cases, 29 scored 209 or more on their first examination; 13 progressed and 12 did not. In four there was insufficient ESD data to determine progression. The positive predictive value was 52% for a diagnosis of dementia (vno progressive dementia) in cases in which the ESD remained above the first percentile of the normal range. Of the rest, 45 scored less than 100 and may be considered to have been demented at the outset. However, two of these subsequently achieved high scores suggesting that they did not have a progressive dementia. In the 65 cases that initially scored between 100 and 208 inclusive, there was insufficient follow up in 19 cases whereas 12 did not progress and 34 did. Overall, in the 116 cases in which a conclusion could be drawn, 24 did not progress and two improved from very low levels whereas 90 did progress for a positive predictive value of 78%. Of the 24 cases that did not progress, only four (16%) have died whereas in the 47 who initially scored 100 or more and progressed, 13 (28%) have died. Twenty eight of the 45 (62%) who initially scored less than 100 have died.
A plot of change in ESD score against time for all the serial observations in all the cases of clinically diagnosed Alzheimer’s disease shows two populations. One group matches that shown in fig 1 and probably contains cases of Alzheimer’s disease. The second group, to the top right of the figure, did not progress with time. These cases are unlikely to have Alzheimer’s disease.
Tables 2 and 3 show the clinical and pathological diagnoses in the 122 necropsy confirmed cases. In table 2, the pathologists’ opinion regarding the role of infarcts in affecting cognition is used and cases with Alzheimer’s disease plus another diagnosis are counted as true positives. Using these criteria, and within a memory clinic, the positive predictive value of a diagnosis of Alzheimer’s disease is 81%. In table 3, the data have been reinterpreted, reclassifying cases with any amount of infarction as mixed dementia when accompanied by histopathological changes of Alzheimer’s disease and reclassifying cases with Alzheimer’s disease plus another diagnosis (other than mixed dementia) as “other dementia.” These cases numbered two with coexisting Parkinson’s disease and 14 with coexisting diffuse Lewy body disease.24 25 This condition was not recognised, clinically or pathologically, during the early part of this study and the diagnoses have been made on retrospective re-examination of the necropsy material. Cases of diffuse Lewy body disease without the changes of Alzheimer’s disease (11 cases) were always classified as “other dementia.” The positive predictive value of a diagnosis of Alzheimer’s disease fell to 44% under these conditions.
Cross tabulation of the clinical and pathological diagnoses using standard criteria
Cross tabulation of the clinical and pathological diagnoses reclassifying cases with changes consistent with Alzheimer’s disease and any amount of infarction as mixed dementia
Comparing the 24 non-progressors with the 47 who initially scored more than 100 on the ESD and progressed, we found no difference between these populations regarding age at onset, delay to clinical evaluation from onset, age at which they were seen, and educational or occupational level. There was a significant sex difference: 16 of 34 men were misdiagnosed compared with only eight of 37 women (χ2=4.6, p=0.033 with Yates’ correction). The patients’ cognitive status at the initial evaluation, measured by their first score on the ESD, did differ (progressors mean 177 (SD 34), non-progressors 206 (SD 23), p=0.0002) and the non-progressors had longer follow-up (1342 (SD 689) days v 1833 (SD 1099) days, p=0.0211). The non-progressors scored more highly on the anxiety and depression components of the HAD,26 but these differences were not significant (depression mean 3.8 (SD 4.6)v 6.1 (SD 4.4), p=0.16; anxiety 5.1 (SD 4.5)v 7.0 (SD 4.3), p=0.197) and three of 14 non-progressors compared with none of 35 progressors had a history of depression (χ2=3.7, p=0.054 with Yates’ correction). There was no difference in the use of antidepressant drugs, past or present use of alcohol, or in the examiner’s subjective rating of the patients’ moods. Abnormalities in the plantar responses, tendon reflexes, and primitive reflexes were of similar frequency in both groups as was the incidence of epilepsy and of leukoaraiosis or infarction on neuroimaging.
Discussion
Although the histopathological criteria for Alzheimer’s disease are not yet settled,6 necropsy is the gold standard for validating the clinical identification of Alzheimer’s disease. In recent series the positive predictive value is typically 80%. We have identified two fundamental faults with these reports which, in combination, reduce the positive predictive value to well below 50% for early Alzheimer’s disease, which has major implications for therapeutic trials.
A crucial question is that of how representative are our results of memory clinic patients in general. We included all the patients seen by VH but only used data from his patients simply because his research commitment in memory disorders meant that his patients were assessed according to the University of Western Ontario Dementia Study protocol rather than on a simple clinical basis. This practice should not be different from any other research based memory clinic. The patients were not otherwise selected and the vast majority of referrals were secondary referrals from the patients’ family physicians. It has also been suggested that our figures show a specific diagnostic deficiency but this suggestion is immediately refuted by the finding that our crude positive predictive value of 81% is closely comparable to data from other large centres; it is only after correction for the fallacies that the figure falls dramatically and we suggest that re-analysis of the data from other centres, after sufficient follow up, would confirm our figures. We think that our data are closely representative of the normal practice of a memory clinic.
The cause of the first fallacy, that of verification bias,10-12 lies in the extrapolation of necropsy material to the whole clinic population. We have shown that the clinical diagnoses in those having necropsy differed substantially from those who did not. Thus necropsy material cannot be extrapolated to the whole memory clinic population. We have also shown that of those who did not have a necropsy, fully 22% did not progress, despite clinical diagnoses of Alzheimer’s disease, vascular dementia, or mixed dementia, and in the presence of sufficient follow up to determine whether progression occurred or not. Although this problem has not been appreciated in previous studies, it should not come as a surprise. Depression, age associated memory impairment, and drug effects may all mimic early Alzheimer’s disease without subsequently progressing. Cases of vascular dementia may not progress if they have no further vascular events, but these are very rare in our memory clinic and would not have significantly affected the results. Furthermore, of the 22% of patients who did not progress, only 16% have died. Conversely, of those with a progressive dementia, between 28% and 62% have died, depending on severity at entry. This differential death rate will cause cases of progressive dementia to be overrepresented in necropsy based data producing a substantial verification bias. Forty per cent of our cases had necropsy, so that the 22% non-progression rate was located in 60% of the whole population. Thus in the whole study population the non-progression rate was about 13%; only 87% of cases diagnosed as having dementia eventually become available for necropsy.
There was a bias towards necropsy in patients with Alzheimer’s disease in our study. We have no definite explanation for this. There were no demographic differences between those who died with and without necropsy that might explain the difference. We are not aware of any other published reports that detail the clinical diagnoses of those who died without necropsy and so cannot comment on whether or not the finding is unique. However, the finding does not invalidate either of the fallacies; the non-progressors did not die and dual diagnosis remains a problem regardless of the true pathological diagnoses in those dying without necropsy.
The cause of the second fallacy, that of dual pathology, lies in the nature of the pathological diagnosis. All previous series,1-4 6-8 13 with one exception of which we are aware,5 have allowed dual pathology such as diffuse Lewy bodies and infarcts to count as a true positive diagnosis of Alzheimer’s disease. Diffuse Lewy bodies have been reported to coexist with Alzheimer’s disease in 1%27 to 30% of patients.4 28 29 Fourteen of our patients (11%) had cortical Lewy bodies with Alzheimer’s disease, and 11 patients (9%) had diffuse Lewy body disease. Some authors consider Alzheimer’s disease coexistent with diffuse Lewy bodies to be a variant of Alzheimer’s disease.28 29 However, not only these authors but others and ourselves22 30 have identified clear cognitive and biological differences between Alzheimer’s disease with and without coexistent diffuse Lewy bodies suggesting that they are different entities and probably represent the coexistence of two common degenerative dementias. The role of infarcts is more problematical for two reasons. Firstly, there is no agreement on the minimum volume of infarction that may affect cognition and so turn a case of Alzheimer’s disease into one of mixed dementia. The original work of Tomlinson,31 who suggested volumes of 50–100 cm3, has been superseded by findings of cognitive impairment in association with volumes as low as 1 cm3.32-37 Secondly, most studies have not described any formal criteria for attributing a role in impairing cognition to infarcts. The previous data reported from our database2 identified a role for infarcts only on the strength of the pathologists’ opinion. Sulkava et al 1 state only that there were “no major vascular lesions.” Joachim et al 4 only considered infarcts larger than lacunes to be relevant. Perry et al 27 thought that ischaemic lesions of less than 20 ml were “minor.” The CERAD study3 mentions “vascular lesions of varying nature and size” with their role being based on the pathologists’ opinion. Similarly, Morris et al 13 reported infarcts in 35% but these affected a “sizable volume” in only one. Molsa et al 5 allowed the presence of any amount of infarction to convert a diagnosis of Alzheimer’s disease to combined dementia, 10% having this diagnosis. Clear data regarding the frequency with which infarcts and Alzheimer’s disease coexist in data based on Alzheimer’s disease are difficult to find because small infarcts may be ignored—for example, in the study of Mendez et al.7 In the consortium to establish a registry for Alzheimer’s disease (CERAD) study3 coexisting infarcts were seen in 28% and Morris et al 13 reported coexistence in 35%, these figures being very similar to our findings. We have found that cases of mixed dementia, including those with only small amounts of infarction, behave differently in their age of onset and neuropsychological progression (Bowler et al, unpublished observations and Bowler et al 38).The balance of evidence is now clearly moving towards a role in cognition for even small infarcts occurring incidentally in patients with Alzheimer’s disease and it would seem unwise to ignore them.
Dual pathology is important in that it reduced the positive predictive value of a diagnosis of Alzheimer’s disease from 81% to 44% in our database, 35 of 76 true positive cases of Alzheimer’s disease being reclassified. Combining the fallacies of dual diagnosis and verification bias together, the overall positive predictive value of a diagnosis of Alzheimer’s disease is 70% allowing Alzheimer’s disease with other conditions and only 38% for pure Alzheimer’s disease, with the bulk of the errors lying in early cases.
Additional tests and review of the clinical aspects to identify phenomena that improve the diagnostic accuracy in early cases are urgently needed. The mean age of onset was 66 and given the higher incidence of diagnostic error in men we speculate that part of the error arises as a result of apparent loss of function after retirement. We had anticipated that depression would be a major factor, but could not confirm this. Neither simple focal neurological signs nor leukoaraiosis or infarcts on neuroimaging distinguish between the groups.
The findings reported here are important as they show two fallacies regarding the necropsy confirmation of the clinical diagnosis of Alzheimer’s disease and revise downwards the diagnostic accuracy for the detection of pure Alzheimer’s disease. Therapeutic trials based on the clinical diagnosis would be at considerable risk of having a real therapeutic effect masked if 62% of cases either do not have Alzheimer’s disease, or have Alzheimer’s disease plus a second disease. Our data also suggest that mixed dementia may be much more common than is currently appreciated39 with implications for treatment of the vascular component and raising the possibility that the recently reported beneficial effect of non-steroidal anti-inflammatory drugs40-43 may be due to an effect on coexistent cerebrovascular disease.
Acknowledgments
VH was a career investigator and JB was a fellow with the Heart and Stroke Foundation of Ontario. Funding for this analysis was provided by the Ministry of Health of Ontario Trillium Clinical Scientist Award to VH.