• carotid artery
• stenosis
• angiography

Comparing a new test with a standard involves measuring disagreement. In the case of measuring carotid artery stenosis, some of the disagreement between different tests is because of inherent differences in how the stenosis is demonstrated (test characteristics). This is what we are most interested in when assessing a new technology. However, some of the disagreement simply reflects variability in how we physically make the measurement with the standard technique. Choosing the point of maximum stenosis, choosing the point in the common carotid artery for use as a denominator, measuring from an eyepiece, or measuring from calipers all introduce variation when measuring carotid stenosis. The resulting observer variability in reporting contributes to disagreement between methods but to some extent is independent of the method used to generate the angiogram in the first place.

In the medical literature, disagreement between methods is often attributed entirely to test characteristics, with little appreciation of the role of observer variability in reporting. When one method is compared with another and disagreements emerge, it is not readily apparent how much of the disagreement is caused by the method used and how much by the process of measurement, unless observer variability data are also presented. In the recent paper from Patel et al, interobserver variability data are presented but their significance in relation to overall agreement does not appear to have been appreciated.¹

Using the data from Patel et al (tables 2 and 4) for symptomatic carotid arteries, it is noted that when 34 carotid digital subtraction angiograms …