J Neurol Neurosurg Psychiatry doi:10.1136/jnnp-2012-304041
  • Cognitive neurology
  • Research paper

The NINDS-Canadian stroke network vascular cognitive impairment neuropsychology protocols in Chinese

  1. Vincent Mok2
  1. 1Department of Psychological Studies, The Hong Kong Institute of Education, Tai Po Hong Kong SAR, China
  2. 2Department of Medicine and Therapeutics, The Chinese University of Hong Kong, Shatin, Hong Kong SAR, China
  3. 3Department of Neurology, Jinling Hospital, Nanjing University School of Medicine, Nanjing, China
  4. 4Department of Radiology and Organ Imaging, The Chinese University of Hong Kong, Shatin, Hong Kong SAR, China
  5. 5Hauenstein Neuroscience Center at Saint Mary's Health Care, Michigan, United States
  6. 6Department of Medicine, University of Toronto, Toronto, Canada
  1. Correspondence to Professor Vincent Mok, Department of Medicine and Therapeutics, 9/F Clinical Sciences Building, Prince of Wales Hospital, The Chinese University of Hong Kong, Shatin, New Territories, Hong Kong SAR, China; vctmok{at}
  • Received 19 August 2012
  • Accepted 29 October 2012
  • Published Online First 18 December 2012


Background and purpose Vascular cognitive impairment (VCI) affects up to half of stroke survivors and predicts poor outcomes. Valid and reliable assessement for VCI is lacking, especially for the Chinese population. In 2005, the National Institute of Neurological Disorders and Stroke and Canadian Stroke Network (NINDS-CSN) Harmonisation workshop proposed a set of three neuropsychology protocols for VCI evaluation. This paper is to introduce the protocol design and to report the psychometric properties of the Chinese NINDS-CSN VCI protocols.

Methods Fifty patients with mild stroke (mean National Institute of Health Stroke Scale 2.2 (SD=3.2)) and 50 controls were recruited. The NINDS-CSN VCI protocols were adapted into Chinese. We assessed protocols’ (1) external validity, defined by how well the protocol summary scores differentiated patients from controls using receiver operating characteristics (ROC) curve analysis; (2) concurrent validity, by correlations with functional measures including Stroke Impact Scale memory score and Chinese Disability Assessment for Dementia; (3) internal consistency; and (4) ease of administration.

Results All three protocols differentiated patients from controls (area under ROC for the three protocols between 0.77 to 0.79, p<0.001), and significantly correlated with the functional measures (Pearson r ranged from 0.37 to 0.51). A cut-off of 19/20 on MMSE identified only one-tenth of patients classified as impaired on the 5-min protocol. Cronbach's α across the four cognitive domains of the 60-min protocol was 0.78 for all subjects and 0.76 for stroke patients.

Conclusions The Chinese NINDS-CSN VCI protocols are valid and reliable for cognitive assessment in Chinese patients with mild stroke.


Cognitive impairment is a major sequalae of stroke.1 Studies in the Chinese populations showed that almost half of stroke patients suffer from varying degree of cognitive impairment poststroke.2–4 Vascular cognitive impairment (VCI) is an umbrella term that encompasses the full spectrum of cognitive impairment related to cerebrovascular disease, from brain-at-risk to Vascular Dementia (VaD).5 VCI is a strong predictor of death6 and a major determinant of functional disability independent of physical impairment.7 Milder forms of VCI may progress to VaD.8 The recognition of the importance of VCI influenced clinical trials for secondary stroke prevention to emphasise cognitive endpoints as outcome measures.9

Because of the lack of generally accepted VCI test batteries, treatment trials on stroke patients had used tests developed based on the cognitive features of AD as cognitive outcome measures.10 However, important neuropsychological differences between AD and VCI have been demonstrated, including more impaired executive functions and slower information processing speed but better episodic memory in VCI patients.11 Therefore, cognitive instruments for AD might not be sensitive to VCI or predictive of functional levels in these patients. To date, there is no consensus in neuropsychological evaluation of VCI, especially among the Chinese population. An ideal VCI neuropsychology protocol should be derived based on established, empirically derived, reliable and valid brain-behaviour relationships.12

To develop a consensus in the disease construct and its study methodology, and to facilitate multi-center, cross-cultural approach for VCI research, in April 2005, the National Institute of Neurological Disorders and Stroke and the Canadian Stroke Network (NINDS-CSN) sponsored a VCI Harmonisation Workshop to develop standardised methodology and minimal dataset in the areas of neuropsychology, neuroimaging, epidemiology, pathology, and experimental research.12 At the Harmonisation Workshop, the Neuropsychological Working Group recommended a set of three VCI neuropsychology protocols. We have modified and adapted the NINDS-CSN VCI protocols into Chinese. The objective of this study is to evaluate (1) the external validity of the Chinese VCI protocols, defined by the ability of these protocols in differentiating stroke patients from non-stroke, cognitively normal controls; (2) concurrent validity, defined by the inter-correlations between protocol summary scores with daily functioning; (3) internal consistency between cognitive domains of the protocol; and (4) ease of administration and clinical feasibility of the protocols in Chinese elderly subjects with a predominately low education level.


Stroke patients

Stroke patients were invited to participate in the study if they aged 50 years or older; had an ischaemic stroke within the past 6–18 months, defined according to NINDS Stroke Data Bank criteria,13 with at least one focal symptom and/or sign of stroke in the acute phase; and had an available informant who is knowledgeable about patient's medical history and cognitive performance and who saw the patient at least three times a week for at least 5 years prior to the recruitment. Stroke patients with history of cerebral haemorrhage, a pre-existing diagnosis of dementia, or if they scored <16 on the Cantonese Mini-Mental State Examination (CMMSE)14 at the study screening, were excluded. Those with prior history of ischaemic strokes were not excluded.

Stroke-free controls

Control subjects were recruited if they did not have a history of stroke or transient ischaemic attack and were cognitively normal, as defined by a CMMSE score ≥26. Efforts were made to match the control and patients based on demographic factors by selecting potential control subjects whose age, gender and education best matched the recruited stroke subjects. To facilitate this matching process, each stroke subject was encouraged to select his or her spouse, family member or friend who met inclusion criteria for entrance into the study.

Common exclusion criteria for controls and patients included current or past history of medical or psychiatric conditions that are known to cause central nervous system or cognitive dysfunctions; sensory or language impediment hindering participation in cognitive testing; and inability to give written informed consent. The study was approved by the Joint CUHK-NTEC Clinical Research Ethics Committee.

MRI protocol

MRI was performed with a 3.0-T system (Achieva 3.0 T TX Series, Philips Medical System, Best, The Netherlands). Diffusion weighted imaging (TR/TE/excitation=2900/66/1, matrix=152×128, Field of view, FOV=230 mm, slice thickness/gap=5 mm/0.5 mm, Echo planar imaging, single -shot factor=90, acquisition time=52.2 s) with three orthogonally applied diffusion gradients (b values of 1000 500, and 0 s/mm2) was used. Other sequences performed included: blood sensitive venous bold sequence (TR/TE/excitation=17/23/1 flip angle of 15°, slice thickness/gap=2 mm/−1 mm, FOV=230×183, matrix=256×203, time of acquisition=2 min 53.8 s); Axial spin echo T1-weighted FFE (TR/TE/excitation=25/2.3/1, flip angle 30° FOV=230 mm, slice thickness/gap=3 mm/−1.5 mm, matrix=256×256, time of acquisition=2 min 43 s), Turbo spin echo T2 –weighted (TR/TE/excitation=2622/80/2, turbo factor of 18, FOV=230 mm, slice thickness/gap=5 mm/0.5 mm, matrix=512×346, time of acquisition=2 mi 21 s), and axial FLAIR (Fluid attenuated inversion recovery; TR/TE/TI/excitation=11 000/125/2800/1, turbo factor of 31, FOV= 230 mm, slice thickness/gap=5 mm/0.5 mm, matrix=352×248, time of acquisition=3 min 18 s) sequences.

MRI pre-processing was performed by a series of operations, such as non-uniformity correction, spatial standardisation, brain extraction. Tissue classification in brain is to separate grey matter, white matter and cerebrospinal fluid from the brain MRI data. These operations were implemented using the Insight Segmentation and Registration Toolkit ( White matter lesions were detected using a fully automated clustering-based quantitative method on the co-aligned FLAIR and T1W MRI data.15 The whole-brain segmentation was achieved using an atlas-based approach16 that automatically segmented brain structures such as hippocampus, lateral ventricle, and amygdala. The intracranial volume was also quantified using this approach. Infarcts were manually delineated on the FLAIR images using the software ITK-SNAP, and its volume was calculated as the number of voxels in the infarct label multiplied by the voxel size in the FLAIR image.

The NINDS-CSN VCI neuropsychology protocols

The NINDS-CSN VCI Neuropsychological Working Group recommended a set of three neuropsychology protocols. The protocols incorporated well-established neuropsychology tests selected based on considerations of psychometric properties (ie, test validity and reliability), clinical utility, feasibility for use in Chinese elderly patients with a predominately low educational background, accessibility, domain specificity for 60-min protocol, cost, lack of ceiling or floor effects, and prior use of test in VCI samples, in particular in the Chinese population. The 60-min protocol was designed for use in studies that require a breakdown of cognitive abilities by domain, so the protocol contains recommended tests in four domains: executive/activation, language, visuospatial and memory. Tests for the 30-min protocol were selected from within the 60-min protocol to be used as a clinical screening instrument for patients. The 5-min protocol was developed for use by primary care physicians, nurses and other health professionals for quick screening in their office or at the bedside, and for very large epidemiologic studies in which sensitivity and ease of administration are especially important. There is a potential application for the 5-min protocol for telephone administration.

The 60-min protocol

Executive/activation domain

Category fluency test17

It requires subject to generate as many animal names as possible in 1 min. It measures speed and activation as well as executive processes including clustering, set-shifting and retrieval.

Symbol-Digit Modalities Test (SDMT)18

This brief timed coding test is a variant of the Digit-Symbol Coding subtest in the Wechsler Adult Intelligence Scale. This test was chosen over the original Digit-Symbol Coding because many Chinese elderly patients with low education level are more familiar with writing Arabic numbers than drawing abstract symbols. Timed coding tasks are sensitive to vascular pathology.19

Colour trails test (CTT)20

The CTT is considered a cultural analogue of the Trail Making Test in the Halstead-Reitan Battery for timed assessment of visuomotor speed and executive functions in VCI.19

Language/lexical retrieval domain

Modified Boston naming test (mBNT)21

The Boston Naming Test is one of the most popular confrontation naming tests. A modified 15-item Chinese version was used.21

Visuospatial domain

Rey-Osterrieth Complex Figure Copy (RCFT)22

It is a brief visuoconstructional task with several standardisation samples available.23 ,24 The standard 36-item scoring method was used.22

Memory domain

Hong Kong List Learning Test (HKLLT)25

The HKLLT is developed based on the model of the California Verbal Learning Test.26 It has been validated in both normal and clinical samples.25

RCFT delayed recall

The total score at the 30-min delayed recall trial was recorded.22

The 30-min protocol

The 30-min protocol is made up of the following subset of test measures in the 60-min protocol: Animal fluency, SDMT and HKLLT.

The 5-min protocol

The 5-min protocol is composed of the five items of the Montreal Cognitive Assessment, Hong Kong version (HK-MoCA), which is a brief screening test sensitive to mild cognitive symptoms in patients with cerebral small vessel disease.27 The 5-min protocol is a 30-point scale with the total score computed as the sum of 5-word immediate recall (5-point), delayed recall (5-point) and recognition (5-point), a 6-item orientation (6-point) and the animal fluency test (0.5 point for each correct exemplar generated, maximum score 9 points) of the HK-MoCA. The 5-min protocol may be further validated for telephone administration. In addition to the 5-min protocol, the total score of the complete HK-MoCA was recorded.

Cantonese Mini Mental State Examination (CMMSE)14

The 30-point Mini-Mental State Examination (MMSE) is one of the most popular general cognitive screening tests.28 A cut-off value of 19/20 yielded sensitivity and specificity of 97.5% and 97.3%, respectively for detecting significant cognitive impairment in the local elderly population.14 The CMMSE and HK-MoCA were administered as supplementary tests in the 60-min and 30-min protocols.

Calculation of protocol summary scores

Individual tests scores were converted to standardised z scores, based on the scores of the control sample. To maintain consistency in the direction of impairment, z scores for CTT time scores were multiplied by −1 such that lower scores denoted poorer performance. The z scores of the individual tests were averaged to form the 60- and 30-min protocols summary scores in order to maintain the relative weight of individual cognitive domains contributing to the protocol summary scores.

The following scales were administered in addition to the neuropsychological battery to measure functional and physical functioning of patients: (1) The Stroke Impact Scale (SIS),29 which is a self-report quality of life instrument for stroke patients that evaluates physical, memory, emotional, and communication functioning sections as well as Activities of Daily Living (ADL) and community participation ratings; (2) Disability Assessment for Dementia (CDAD),30 that measures both basic and instrumental activities of daily living and leisure activities in patients with cognitive impairment and dementia; and (3) National Institute of Health Stroke Scale (NIHSS) for measurement of stroke severity at admission. A trained psychometrician with a psychology background administered the tests to the subjects under the supervision of a psychologist.

Statistical analysis

Comparison of demographic and clinical variables between stroke patients and control subjects were performed using the independent t test for continuous variables and the χ2 test or Fisher's Exact Test for categorical variables. Because the patient group consisted of a higher proportion of male and was significantly less educated than the control group, group comparisons of the individual test scores and protocol summary scores were performed with analysis of covariance with gender and education adjusted. The Cohen's d statistic was used to index the effect size of individual tests of the protocols.

External validity

External validity of the protocols was determined by the ability of protocol summary score in differentiating stroke patients from control subjects. Using the receiver operating characteristics (ROC) curve analysis, the area under curve (AUC) indicates the probability that a randomly chosen subject would be ranked correctly by each protocol summary score. AUC varies from 0.5 (no accuracy) to 1 (perfect accuracy).31

Furthermore, to assess the additional value of the 5-min protocol and the HK-MoCA over the CMMSE as screening instrument, the number of stroke patients classified as impaired by CMMSE was compared against that of the HK-MoCA and 5-min protocol. Cut-off values for impairment were 19/20 and 21/22 for CMMSE and HK-MoCA, respectively. These values were derived empirically in local Chinese elderly samples with a predominately low education background.14 ,27 Because no parallel data was available for the 5-min protocol, impairment was categorically defined as z<−1.5 from control mean25 ,32 (ie, 1.5 SDs below control mean). Test agreement was measured using the Kappa statistic.

Concurrent validity with functional measures and internal consistency

Concurrent validity of the test protocols was examined by Pearson correlation between the protocol summary scores and SIS memory section score and total CDAD score. Internal consistency of the four cognitive domains of the 60-min protocol was measured using the Cronbach's α statistic.


Fifty stroke patients and 50 controls were recruited. Mean NIHSS of stroke patients at admission was 2.2 (SD=3.2). The mean number of days from onset of index stroke to cognitive assessment was 227.7 (SD=63.0). A brief description of neuroimaging features of stroke patients is provided in table 1.

Table 1

Measures of infarct and white matter lesions of stroke patients

Compared to controls, stroke patients consisted of a higher proportion of male (χ2(1, N=100)=4.89, p=0.27), were less educated (t(98)=3.36, p=0.001), had higher prevalence of vascular risk factors including hypertension (χ 2(1, N=100)=17.24, p<0.001), diabetes mellitus (χ2(1, N=100)=5.38, p<0.05), hyperlipidaemia (χ2(1, N=100)=36.68, p<0.001) and atrial fibrillation (χ2(1, N=100)=12.95 p<0.001). Group comparisons of demographic and clinical data are summarised in the table 2.

Table 2

Group comparison of demographic and clinical variables

Stroke patients scored significantly lower than controls on all neuropsychological measures, with effect size between 0.7 and 1.7 for individual tests and between 1.2 and 1.4 on protocol summary scores. Group comparisons of neuropsychological performance are presented in table 3.

Table 3

Mean, standard deviation and effect size of neuropsychological measures

External validity

AUC (95% CI) of protocol summary scores are as follows: 60-min protocol 0.79 (0.69–0.89), 30-min protocol 0.79 (0.69 to 0.89), and 5-min protocol 0.76 (0.66–0.87), p<0.001 for all models. AUC for the complete HK-MoCA and CMMSE are 0.75 (0.63–0.86) and 0.79 (0.68–0.89), p<0.001, respectively. The figure 1 shows the ROC curves of the three protocol summary scores.

Figure 1

Receiver operating characteristics of Chinese NINDS-CSN vascular cognitive impairment protocol summary scores in differentiating stroke patients from controls.

Using established cut-off values, 6% of patients were classified as impaired on the CMMSE, whereas 60% were impaired on the HK-MoCA. A cut-off of 19/20 on the CMMSE identified only 10.0% of patients who had impaired score on HK-MoCA and 9.4% who scored z<−1.5 on the 5-min protocol. Kappa statistic reported was 0.082 (p=0.15) between HK-MoCA and CMMSE, and 0.069 (p=0.18) between 5-min protocol and CMMSE.

Concurrent validity and internal consistency

All protocol summary scores significantly correlated with SIS Memory Domain and CDAD total. Table 4 shows the intercorrelations between neuropsychological performance and functional measures. Cronbach's α across the four cognitive domains of the 60-min protocol was 0.78 for all subjects and 0.76 for stroke patients.

Table 4

Intercorrelations between protocol summary scores and functional measures


In this study we showed that all the three Chinese NINDS-CSN protocols are valid in differentiating stroke patients from healthy controls and significantly correlated with daily functioning in stroke patients. Internal consistency of the 60-min protocol was also acceptable.

In our study, large effect size was observed for both individual tests and protocol summary scores,33 suggesting that the four neuropsychological domains selected were appropriate for VCI assessment. As neuropsychological sequelae of vascular insults depend on the location of insults in accordance to models of brain-behavior relationships, one major advantage of the 60 and 30-min protocols is the ability to yield a breakdown of performance by cognitive domain within a reasonably short time of assessment, which is not afforded by simpler screening tests such as the MMSE and MoCA. As for screening purpose, the 5-min protocol performed similarly well with the 30- and 60-min protocols in differentiating subject groups and in predicting functional status of stroke patients. Despite that the CMMSE also performed similarly as the test protocols in the ROC analysis, it identified only about one-tenth of patients considered impaired on the 5-min protocol and HK-MoCA. Given the high CMMSE observed in the stroke sample (M=25.2 (SD=3.3)), this finding emphasises the need of a more sensitive test in addition to the traditional MMSE to assess stroke or mild cognitive symptoms,27 ,34 and our results support the use of the 5-min protocol to complement the MMSE for VCI screening. Potential telephone administration of the 5-min protocol is likely to extend its application in large epidemiological studies and telemedicine, making cognitive screening more readily accessible and less costly.

In terms of ease of administration and feasibility, we found that the protocols were generally well accepted by elderly Chinese subjects. Most stroke patients were able to complete the main protocols plus the supplementary tests in 1–1.5 h without experiencing significant fatigue. However, we observed that illiterate subjects were less motivated to perform paper-and-pencil tests such as CTT and RCFT. The mean education of our stroke patient was 5.9 years, which was much lower than that in Caucasian stroke samples.35 Nevertheless, given the improved accessibility to formal education post war and implementation of compulsory education in Hong Kong in the 1970s, we anticipate an improvement on the applicability of paper-and-pencil tests to local stroke patients of future cohorts.

There are several limitations in the current study. First, the stroke sample suffered relatively mild stroke and cognitive impairment, and those with significant language and motor impairment were excluded. Therefore, our results may not be generalised to patients with more severe strokes or cognitive impairment. Likewise, we did not examine patients in the ‘brain-at-risk’ stage of VCI, those with subclinical brain lesions without clinical strokes, or patients with VaD.5 Therefore, our sample could not be considered representative of the entire spectrum of VCI. Second, our study did not evaluate the test-retest and inter-rater reliability of the test protocols as the individual tests selected had already demonstrated well-established psychometric properties. Third, despite our strategy for group matching, group differences in gender and education level existed. Nevertheless, such effects were adjusted in the statistical analysis. Fourth, the strategy we used in recruiting the spouse of patient as control might have introduced possible bias due to effects of shared environmental risk factors (such as diet and smoking) for VCI that could have minimised group differences in protocol measures. Fifth, even though we used a high cut off score (≥26) on the CMMSE for screening of cognitively normal controls, we could not completely rule out subjects with mild cognitive impairment in this group. Likewise, our a priori selection of patients and controls using different cognitive criteria might be a compromise to the aims of the NINDS-CSN Harmonisation workshop that emphasises ground up development of VCI research based on valid brain-behaviour relationships.12 Analyses of concurrent validity of the protocols using detailed neuroimaging measures in the whole study sample are currently underway and will be reported in the future.

In conclusion, this study demonstrated that the three Chinese NINDS-CSN VCI protocols are valid and reliable for use for cognitive assessment in Chinese stroke patients. These results encourage further development of the NINDS-CSN protocols in different cultures and VCI subgroups, as recommended by the Harmonisation workshop.12 We believe that this study represents an important step in cross-cultural VCI research, as standardisation of neuropsychological protocols and study methodology allows VCI researchers from different cultures to communicate using a common language. It also allows the development of a common dataset geared towards better understanding of VCI in general, and to the identification of cultural mediators important for the management of individual patients in the clinic.


  • AW and Y-y X contributed equally

  • Funding This study was supported by the Health and Health Services Research Fund (Reference number: 06070231).

  • Contributors AW: study design, data collection, statistical analysis, manuscript preparation. Y-yX: MRI analysis, data collection, statistical analysis, manuscript preparation. DW: MRI analysis. SL: MRI analysis. WWCC: MRI acquisition and analysis. PK: data collection, statistical analysis. DN: study design, adaptation of Chinese neuropsychology protocols. SEB: study design, design of MRI protocols, adaptation of Chinese neuropsychology protocols. KSLW: study design. VM: study design, clinical assessment, manuscript preparation.

  • Conflicts of interest None.

  • Provenance and peer review Not commissioned; externally peer reviewed.


Visit the full archive of podcasts for JNNP here >>

Free sample
This recent issue is free to all users to allow everyone the opportunity to see the full scope and typical content of JNNP.
View free sample issue >>

Don't forget to sign up for content alerts so you keep up to date with all the articles as they are published.