Transgenic mice expressing human APOE-epsilon4 develop an age-dependent decline in memory without pathological features of Alzheimer's disease (AD). This implicates APOE in the maintenance of memory during normal senescence, but parallel human studies are limited because longitudinal investigations of memory usually do not exclude patients with AD or "questionable" AD (QD). The current study examined the effect of APOE on cognitive function over time in elderly without dementia. We hypothesized that, compared to other APOE alleles memory decline even in healthy elderly would be greater among those with an APOE-epsilon4. The results of neuropsychological tests, grouped into domains of memory, language and visuospatial/cognitive function by factor analysis, were examined at three intervals over a seven-year period in 563 healthy elderly without AD or QD using generalized estimating equations. Memory performance declined over time, while scores on the visuospatial/cognitive and language factors did not change. Increased age was associated with lower scores, and higher education with higher scores on all factors at each interval. No APOE allele was associated with performance on a specific cognitive factor at any interval, but the presence of an APOE-epsilon4 allele was associated with a more rapid decline in the memory factor over the follow-up period. The effect was most pronounced among individuals with less than 10 years of formal education. There was no similar time-dependent relationship between APOE-epsilon4 and the language or visuospatial/cognitive factors. Transgenic mice and elderly humans without AD or QD expressing APOE-epsilon4 show a decline in memory performance over time. These observations provide evidence for an APOE-specific effect on memory during senescence.