Working memory (WM), the process by which information is coded into memory, actively maintained and subsequently retrieved, declines with age. To test the hypothesis that age-related changes in prefrontal cortex (PFC) may mediate this WM decline, we used functional MRI to investigate age differences in PFC activity during separate WM task components (encoding, maintenance, retrieval). We found greater PFC activity in younger than older adults only in dorsolateral PFC during memory retrieval. Fast younger subjects showed less dorsolateral PFC activation during retrieval than slow younger subjects, whereas older adults showed the opposite pattern. Thus age-related changes in dorsolateral PFC and not ventrolateral PFC account for WM decline with normal aging.