Research in Parkinson's disease has given little attention to the reliability of subjective-qualitative assessment, or to the empirical aggregation of physical signs to supraordinate indexes of motor disturbance. We illustrate methods for examining observer reliability and discuss the importance of reliability to the interpretation of results. Observers rated patients (n = 70) for physical signs (selected Columbia Scale scores), disability stages, and abnormal involuntary movements. Observer agreement was achieved for Columbia scores and disability stages (range of Spearman rho = 0.67-0.95), but not for dyskinesias or dystonic postures. A factor analysis of Columbia scores revealed two indexes of motor disturbance: motor deficiency and tremor.