Objectives: To evaluate the accuracy of routinely available non-invasive tests (spiral computed tomographic angiography (CTA), time of flight magnetic resonance angiography (MRA), and colour Doppler ultrasound (DUS)), individually and together, compared with intra-arterial digital subtraction angiography (DSA) in patients with symptomatic tight carotid stenosis; and to assess the effect of substituting non-invasive tests for DSA on outcome, interobserver variability, and patient preference.
Methods: Patients referred from a neurovascular clinic were subjected prospectively to DUS imaging. The operator was blind to symptoms. Patients with a tight carotid stenosis on the symptomatic side were admitted for DSA. CTA and MRA were performed during the admission. The CTA, MRA, and DSA films were each read independently by two of six experienced radiologists, blind to all other data.
Results: 67 patients were included (34 had all four imaging procedures). DUS, CTA, and MRA all agreed with DSA in the diagnosis of operable v non-operable disease in about 80% of patients. CTA tended to underestimate (sensitivity 0.65, specificity 1.0), MRA to overestimate (sensitivity 1.0, specificity 0.57), and DUS to agree most closely with (sensitivity 0.85, specificity 0.71) the degree of stenosis as shown by DSA. When using any two of the three non-invasive tests in combination, adding the third if the first two disagreed would result in very few misdiagnoses (about 6%). MRA had similar interobserver variability to CTA (both worse than DSA). Patients preferred CTA over MRA and DSA.
Conclusions: DUS, CTA, and MRA all show similar accuracy in the diagnosis of symptomatic carotid stenosis. No technique on its own is accurate enough to replace DSA. Two non-invasive techniques in combination, and adding a third if the first two disagree, appears more accurate, but may still result in diagnostic errors.
- CT angiography
- MR angiography
- intra-arterial angiography
- Doppler ultrasonography
- carotid stenosis
- CTA, spiral computed tomographic angiography
- DSA, intra-arterial digital subtraction angiography
- DUS, colour Doppler ultrasound
- ECST, European carotid surgery trial
- MRA, time of flight magnetic resonance angiography
Statistics from Altmetric.com
- CTA, spiral computed tomographic angiography
- DSA, intra-arterial digital subtraction angiography
- DUS, colour Doppler ultrasound
- ECST, European carotid surgery trial
- MRA, time of flight magnetic resonance angiography
Correct diagnosis of the degree of carotid stenosis, in combination with clinical findings, is the key to deciding whether patients require carotid endarterectomy.1 Digital subtraction intra-arterial angiography (DSA) is the reference standard because it was the method used to measure carotid stenosis, and hence stroke risk and the balance of risk and benefit of endarterectomy in the carotid endarterectomy trials.1 However, it has significant inherent risks (1% mortality and 4% disabling stroke in patients with symptomatic ischaemic cerebrovascular disease), so an alternative non-invasive method would be preferable.2 Doppler ultrasound is used as the sole imaging test before endarterectomy in some centres, but there is concern about its reliability. A less operator dependent, reliable, confirmatory non-invasive test is needed to improve confidence in correct patient selection for endarterectomy. Magnetic resonance angiography (MRA) and computed tomographic angiography (CTA) are promising alternatives to DSA, but, as many studies of their accuracy were done in patient cohorts with relatively mild stenoses, adequate data are lacking on their accuracy in patients with symptomatic tight carotid stenoses.3–5
To assess the value of the non-invasive tests in a relevant patient group, we compared the accuracy of CTA, MRA, and DUS—using DSA as a reference standard—in typical patients known to have a symptomatic tight carotid stenosis and being considered for carotid endarterectomy. We then determined whether combining these tests improved diagnostic accuracy, and we also investigated the effect on outcome if DSA were to be replaced by the non-invasive tests. In addition, we examined interobserver variability and obtained the patients' opinions of the tests.
Patients with symptomatic ischaemic cerebrovascular disease who were suitable for carotid endarterectomy were seen prospectively in our neurovascular clinic by a stroke physician or neurologist and assessed clinically. They were examined with Doppler ultrasound (DUS), the ultrasonographer being blinded to the side and nature of the symptoms. A detailed report was completed at the time of examination, documenting the degree of stenosis from velocity criteria and arterial appearance.6 Those with a tight stenosis on DUS (70–99% reduction in lumen) in the symptomatic carotid artery were admitted for confirmation of this with DSA. CTA and MRA were also performed during the admission if the patient gave consent. Thus the study population consisted of all patients known to have at least one symptomatic tight carotid stenosis as demonstrated by ultrasound during a one year period.
CTA and MRA were performed within about 24 hours of DSA. The time between DUS and the other investigations varied depending on bed availability. All imaging equipment was less than three years old.
DUS was performed by one of three operators (two consultant neuroradiologists and one neuroradiographer) on an Acuson 128XP10v with a 7.5 MHz probe (Acuson Inc, Mountainview, California, USA). The peak systolic and end diastolic velocities were recorded routinely from the internal and common carotid arteries and the vertebral arteries, and the peak systolic velocity from the external carotid arteries. The degree of stenosis was determined by comparison with standard velocity tables6 in conjunction with the images of the arteries, all documented and reported at the time of examination.
Digital subtraction intra-arterial arteriography
DSA was done with a GE Advantex system (GE Medical Systems, Milwaukee, Illinois, USA). Both common carotid arteries were selectively catheterised using the Seldinger technique by the femoral approach. Three standard views (lateral and bilateral 45° oblique) of the carotid bifurcation were obtained, with additional views in ambiguous cases. For DSA, stenosis measurements were made on the projectional image showing the maximum stenosis.
CTA was performed on one of two spiral CT scanners (Somatom Plus and GE CTi scanners, Siemens, Erlangen, Germany). Following a test bolus to determine the time to optimal contrast enhancement, 100 ml of non-ionic contrast (Niopam 300 mg/ml) was injected at 3 ml/s, and the neck scanned from C2 to C7 using 3 mm collimation (pitch 1.0, small FOV (field of view), 120 kV, 260 mA). Axial images were reconstructed at 1 mm increments throughout the scan volume. All axial images were hard copied, with additional maximum intensity projection (MIP) and surface shaded display (SSD) images obtained in the majority of cases. For CTA, measurements were made on magnified axial images at the level of maximum stenosis.
MRA was performed on one of two systems (Siemens Magnetom 63SP 1.5T and GE Impact 1.0T). Both two dimensional time of flight (TOF) and three dimensional TOF sequences were obtained at the level of the carotid bifurcations. Twelve 15° MIP images of each carotid bifurcation were generated for each sequence and printed out to hard copy film. For MRA, measurements were made on the projectional MIP image showing the maximum stenosis. If a flow gap was present at the point of maximum stenosis, then “flow gap” was recorded.
The DSA, CTA, and MRA imaging were each read by two of a group of six radiologists experienced in the assessment of the carotid bifurcation (four consultant neuroradiologists, one consultant radiologist in cross sectional imaging, and one neuroradiology fellow). Each mode was read by the two observers, blind to all other clinical and imaging data and to each other's results. The presence of any carotid disease was noted and the maximum percentage stenosis measured, using the common carotid artery (CCA) diameter at a non-atheromatous level as the denominator and the narrowest stenosis as the numerator to calculate the percentage stenosis (this has been shown previously to be the most reliable method for determining the degree of stenosis on DSA).7 All measurements were made on hard copy images using a watchmaker's high resolution measuring eye piece.
All data were entered into a database (Microsoft Access) and used to compute the interobserver variability of CTA, MRA, and DSA. Flow gaps on MRA were coded as 90% stenosis. We could not determine interobserver variability prospectively for DUS in this series as there was only one operator per patient. However, this was determined in a separate study.8 For the comparison of CTA, MRA, and DSA, where there was more than a 10% difference between the observers' readings of a particular scan the imaging was reassessed by the two observers and a consensus value for the percentage stenosis obtained. For other less discrepant values, the average of the readings by the two observers was calculated. The values from DSA were taken to be the reference standard.
In patients who underwent all three non-invasive proceedings as well as DSA, we compared the MRA, CTA, and DUS readings with the DSA readings by two methods. First, the data were dichotomised into operable carotid disease (80% to 99% stenosis)1 and non-operable carotid disease (0–79% stenosis, or occlusion) and analysed using the κ statistic, which assesses agreement beyond that expected by chance alone. A κ of 0.8 to 1.0 indicates excellent agreement, 0.6 to 0.79 good, 0.46 to 0.59 moderate, 0.2 to 0.39 poor, and 0 to 0.19 no agreement. Symptomatic arteries were analysed separately from the contralateral asymptomatic arteries, as we were interested in the accuracy per patient, not per artery. Second, the stenosis values for CTA, MRA, and DUS were compared as continuous variables with DSA using scatterplots. The interquartile ranges of the differences between the two observers for each procedure were also calculated.
To determine the effect of substituting non-invasive tests for DSA on outcome, we then used the European carotid surgery trial (ECST) graph1 of expected change in life expectancy following endarterectomy (which is split by age, sex, and percentage stenosis) to calculate the effect on life expectancy of performing CTA, MRA, and DUS instead of DSA. If the non-invasive procedure could have changed the decision on whether a patient underwent endarterectomy (according to DSA), then the change in life expectancy resulting from that decision was noted.
We also assessed the accuracy of a strategy of using two non-invasive tests combined compared with only one in the patients who had all four procedures.
Following discharge after DSA, the patients were sent a postal questionnaire (appendix) asking about which clinical procedure (CTA, MRA, or DSA) they most and least preferred. Their opinion of DUS was not sought as it was considered to be a standard part of their carotid assessment. Also, DUS was performed some weeks before CTA/MRA/DSA so it would have been more difficult to remember.
Seventy three patients were entered into the study after Doppler examination, but only 67 (92%) actually had DSA. Fifty nine (81%) had CTA and 47 (64%) had MRA (fig 1). Thirty four patients (47%) had all four imaging procedures. Reasons for not performing the procedures are shown in table 1. The most likely procedure to be missed because of patient refusal or contraindications was MRA (10%), followed by DSA (5%); CTA (3%) was the least likely not to be performed. Adverse reactions occurred during 10% of DSA procedures, but not during any other procedure (table 1). These accounted indirectly for a further 6–10% of patients not undergoing MRA or CTA.
The median time lapse between DUS and the other three imaging techniques was 33 days (range 27 to 185 days), whereas CTA and MRA were all performed within 24 hours of DSA.
In the symptomatic arteries, the two observers for DSA agreed with each other's categorisation of operable/non-operable in 79% of patients (κ = 0.6). For MRA it was 79% (κ = 0.52) and for CTA, 76% (κ = 0.48) (table 2). The interquartile ranges for the differences between the two observers were: −0.08 to +4.78 for DSA, −4.08 to +10.57 for MRA, and −0.60 to +9.70 for CTA (fig 2A). In the asymptomatic arteries, there were only three with stenosis of 80% to 99% (that is, operable had they been symptomatic) and therefore the analysis using the κ statistic was very unreliable (table 3 and fig 2B). We did not test intraobserver agreement for CTA, MRA, or DSA in the present study for several reasons—it is unlikely to be worse than interobserver agreement, it would have doubled the amount of work, and it would have been difficult to blind adequately.
The interobserver agreement for DUS was assessed in a separate substudy and is reported in full elsewhere.8 In this substudy, the κ for operable v non-operable stenosis for the two observers who examined the most patients with Doppler ultrasound (n = 206) was 0.62—comparable with the agreement between two observers for the operability of symptomatic stenosis, and better than that for asymptomatic stenosis on DSA in the present study (see below). On DUS, the median difference in per cent stenosis between the two observers was 0%, the interquartile range was also zero, and the minimum and maximum differences were −55% and +65%, respectively. The population in the DUS observer study,8 however, included many fewer patients with operable stenosis (11/206, 5%) than in the present study (20/35 on the symptomatic side (57%)), and more similar to our findings on the asymptomatic side (3/34, 9%). We have not routinely assessed intraobserver reliability on DUS owing to the problem of blinding the observer to the result of their first examination.
Imaging accuracy – individual tests
In the symptomatic arteries, DUS agreed with the DSA categorisation of operable/non-operable in 79% of patients (table 4), MRA agreed in 82%, and CTA agreed in 79%. Most discrepancies in the assessment of eligibility for endarterectomy occurred in patients with 75–80% stenosis on DSA (fig 3A). Compared with DSA, CTA underestimated the degree of stenosis (sensitivity 0.65, specificity 1.0), MRA tended to overestimate the degree of stenosis (sensitivity 1.0, specificity 0.57), and DUS fell between the two (table 4 and fig 3A).
In the asymptomatic arteries, there was a greater proportion of lesser degree stenosis, and the few operable stenoses (had they been symptomatic) rendered the sensitivity calculations unreliable (table 5 and fig 3B).
If the decision to proceed to endarterectomy for symptomatic carotid stenosis were based purely on the degree of stenosis without taking any clinical features into consideration (which would of course not happen in clinical practice), then using CTA instead of DSA would have resulted in no unnecessary operations but seven missed operations (21% inappropriate management). For DUS, four unnecessary operations would have been performed, and three operations missed (21% inappropriate management), and for MRA, six unnecessary operations would have been performed and no operations missed (18% inappropriate management).
However, if risk factors for stroke after TIA other than just the degree of stenosis are taken into account (that is, the patient factors of sex and age), as defined by the ECST graph,1 15% of patients having DUS, 12% of patients having MRA, and 21% of patients having CTA would have received inappropriate treatment. This translates into a change in outcome (life expectancy) of one month lost with DUS and MRA, and two months lost for CTA.
Imaging accuracy – combinations of tests
Finally, the accuracy of using two non-invasive tests in combination was assessed in the 34 patients who had all four imaging tests. CTA disagreed with MRA in the categorisation of operable/inoperable in 13 patients (38%) and with Doppler in 14 patients (41%); MRA disagreed with Doppler in nine patients (26%). However, if any two of the three non-invasive procedures were performed on every patient, with a third non-invasive procedure used if the first two disagreed, then there would only be a disagreement with DSA in two patients (6%).
Patients most preferred CTA, with DSA and MRA being equally less popular (table 6). Of the 44 patients who underwent more than one imaging procedure, 45 (76%) preferred CTA, 15 (22%) preferred DSA, and 10 (21%) preferred MRA. Opinion on DUS preference was not sought because of the time lapse from DUS examination to hospital admission for angiography and because it was an integral and unavoidable part of patient screening.
In this prospective study of a typical population of sequentially referred patients with symptomatic cerebrovascular disease and tight carotid stenosis, who were being considered for endarterectomy, DUS, CTA, and MRA all performed similarly in the measurement of carotid stenosis. CTA underestimated the stenosis, MRA overestimated it, and Doppler lay somewhere in between the two. None of the techniques was highly accurate, particularly around the 70–80% stenosis range where greatest discrimination is needed. We do not know whether the loss of accuracy with non-invasive tests resulting in a few inappropriate endarterectomy decisions may be offset by avoidance of the risk of angiography itself. However, based on the findings in this study, CTA and MRA are inappropriate replacements for DSA so far. Further technical improvements in CTA (for example, multislice CTA9) and MRA (contrast enhanced three dimensional MRA10) have been described, but these still need rigorous assessment against the reference standard, DSA.
The complications of DSA—particularly cardiovascular and peripheral vascular complications, some of which were serious—were in line with previous observations in similar populations.2 This reflects the high prevalence of co-morbidity in a typical patient population encountered in clinical practice, and emphasises the importance of continuing to seek an alternative non-invasive diagnostic test.
The ECST tables for estimating risk and benefit of carotid endarterectomy are based on measurement of stenosis from DSA and were derived from patients who had survived DSA and were randomised into the trial—that is, the risk of DSA was not included in the risk to benefit equation for endarterectomy. The greater safety of non-invasive imaging will result in an overestimation of the change in life expectancy (outcome) when using the non-invasive test instead of DSA. However, this difference is likely to be small, and in any case in the present study the inaccuracy of the non-invasive tests would result in loss of life expectancy. Their greater safety might result in net gain in life expectancy, but this could only be determined in a large randomised trial.
The interobserver variability between paired observers for each imaging mode was similar for CTA, MRA, and DSA and for DUS in our previous study.8 This variability has significant implications with regard to the reliability of individual measurements of stenosis when using any mode, and puts into question the use of a single measurement of stenosis on DSA as the reference standard. Our results are similar to those obtained in two previous studies of the interobserver variability of measurement of stenosis from DSA, which found κ values indicating moderate to good agreement.11,12 We did not measure intraobserver variability, for the reasons given in Results, but it is likely to be no worse than the interobserver variability.
Our analysis does not really compare like with like: CTA and MRA were both assessed by two observers and discrepancies discussed to produce a final agreed value, whereas DUS was performed once only with no second observer, as well as there being a time lapse to DSA during which the carotid disease could truly have changed. It was not practical to repeat the DUS owing to lack of time in a busy clinic. This might account for some of the apparent discrepancies in the degree of stenosis compared with DSA. Where there was a significant delay between DUS and DSA, there was potential for true stenosis progression or occlusion. However, despite the study design allowing several reasons for DUS to appear less accurate than CTA or MRA, it performed equally well.
A sizeable proportion (53%) of patients admitted to the study on an “intention to image” basis did not complete all imaging. However, this reflects the reality of investigating patients with multiorgan disease with complex neuroradiological techniques in clinical practice. In addition to their carotid disease, many of the patients had ischaemic heart disease, peripheral vascular disease, or chronic obstructive airways disease. Indeed six of the 73 patients admitted to hospital for the key test of DSA did not have it, and there was a significant number of complications following DSA which contributed to the number of patients who did not undergo CTA or MRA.
Numerous previous studies of MRA,14 and increasingly of CTA,15 report high sensitivities and specificities for assessing carotid stenosis. However, many studies had methodological problems that may have led to overestimates of accuracy.3–5 We have documented all our cases, rather than excluding from mention those who did not complete all tests. We focused on patients with known symptomatic tight carotid stenosis, the population in whom a confirmatory non-invasive test needs to be most accurate. Many other larger studies have included substantial proportions of subjects with a lower incidence of disease,3,5 which may have led to an overestimation of accuracy.
Our study shows that in a population of patients with a high prevalence of tight symptomatic carotid stenosis, the use of DUS, CTA, or MRA alone instead of DSA may result in clinically important differences in patient treatment. Further studies are required to assess whether a combination of tests, or the new technical advances in MRA and CTA, are sufficient to replace DSA.
This work was supported by grants from the Scottish Office Chief Scientist Office (grant No K/OPR/15/8/F19) and Chest Heart and Stroke Scotland. JMW and SCL were funded by the UK Medical Research Council under the Clinical Research Initiative in Clinical Neurosciences and SCL also by the Scottish Office Chief Scientist Office.
If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.