Our objective was to determine the pattern and extent of generalized and focal neocortical atrophy that develops in patients with epilepsy and the factors associated with such changes. As part of a prospective, longitudinal follow-up study of 122 patients with chronic epilepsy, 68 newly diagnosed patients, and 90 controls, serial magnetic resonance imaging scans were obtained 3.5 years apart. Image subtraction was used to identify diffuse and focal neocortical change that was quantified with a regional brain atlas and a fully automated segmentation algorithm. New focal or generalized neocortical volume losses were identified in 54% of patients with chronic epilepsy, 39% of newly diagnosed patients and 24% of controls. Patients with chronic epilepsy were significantly more likely to develop neocortical atrophy than control subjects. The increased risk of cerebral atrophy in epilepsy was not related to a history of documented seizures. Risk factors for neocortical atrophy were age and multiple antiepileptic drug exposure. Focal and generalized neocortical atrophy commonly develops in chronic epilepsy. Neocortical changes seen in a quarter of our control group over 3.5 years were likely to reflect physiological changes. Our results show that ongoing cerebral atrophy may be widespread and remote from the putative epileptic focus, possibly reflecting extensive networks and interconnections between cortical regions.