Introduction

Assessing the complexity of rehabilitation needs presents a considerable challenge throughout the world. In the US, Canada, Australia and many parts of Europe, classifications of rehabilitation complexity have relied on physical dependency (measured by the Functional Independence Measure (FIM)i 1 or Barthel Index (BI)2) as a surrogate for rehabilitation needs.3 4 These classifications may work reasonably well where patients are medically stable, and physical independence is the main target of intervention. However, they do not capture needs for medical or specialist nursing care; nor do they specifically address the need for cognitive, behavioural or other psychological interventions.

The Northwick Park nursing Dependency Scale (NPDS)5 6 and the equivalent Therapy Dependency Assessment tool (NPTDA)7 8 provide a detailed evaluation of requirements for basic care, nursing, therapy and medical needs, in terms of both the disciplines involved and the proportion of time spent on different rehabilitation activities (including cognitive/behavioural interventions and family support). A common underlying principle of these instruments is that they are designed to be applied either prospectively to measure ‘needs’ for care/intervention, or retrospectively to describe the level of intervention actually provided (ie, what the patient ‘gets’),9 so that in future they could be applied as a framework for quantifying gaps in service provision. However, they are somewhat time-consuming to administer and may not be practical for routine application in services with a high throughput of cases.

The Rehabilitation Complexity Scale (RCS) has been similarly designed to provide a simple measure of the complexity of rehabilitation needs and/or interventions, which is timely to apply and takes account of basic care, specialist nursing, therapy and medical interventions. A preliminary exploration10 of the RCS (version 1) demonstrated that it is simple and practical for routine use across a range of specialist rehabilitation services. In a multicentre cross-sectional analysis,10 it showed clear differences between tertiary (or ‘complex specialised’) and secondary (or ‘district specialist’) rehabilitation services, on the basis of their relative proportions of complex cases (and the staffing levels to cope with them). Clinicians reported favourably on utility, content and face value but noted some ceiling effects for patients with very complex needs. The therapy subscale, which recorded total hours of therapy intervention,ii was found to be difficult to rate prospectively. The instrument was revised to form the RCS version 2, in which the care, nursing and medical scales remain the same, but the therapy scale has been divided into two subscales reflecting (a) the number of therapy disciplines and (b) the overall intensity of treatment.

The term ‘clinimetrics’ was initially coined by Alvan Feinstein in the 1980s11 in recognition of the fact that items within rating tools developed in clinical settings are often chosen for their clinical relevance, rather than their measurement or scaling properties. Scale development is therefore driven by clinical content, which is valued over unidimensionality—often the principal concern of psychometrics. Nevertheless, it is important to understand the extent to which a given instrument can be used as a ‘measure,’ as opposed to simply an ‘assessment tool.’12

The items in the RCS were chosen for their content value, on the basis that needs for care, nursing, therapy and medical input are the principal ‘causes’ of case complexity which (together with length of stay) will ultimately determine the cost of providing a rehabilitation programme for a given individual. Item levels are broadly ordinal, but were designed to reflect clinically important features that may influence care planning, rather than being chosen for their interval qualities. Item scores are expected to be in some way cumulative, but it is pertinent to know whether they may reasonably be ‘summed’ to a total score as an overall indicator of caseload complexity and, if not, whether they can be grouped or should be reported individually.

The aim of the present paper is to report on the key clinimetric properties of the RCS version 2—that is, its reproducibility, validity, feasibility, responsiveness and interpretability13—and to investigate its performance in a sample of patients with highly complex rehabilitation needs. We also explored its dimensionality and relationship with the other rehabilitation dependency scales (the NPDS and the NPTDA), as well as with the BI and the FIM, in order to evaluate its potential as a measure of caseload complexity in complex neurological rehabilitation settings.

Methods

Setting

The Regional Rehabilitation Unit at Northwick Park provides a tertiary postacute rehabilitation service for younger adults with severe complex neurological disabilities, including physical, cognitive, behavioural and/or communicative problems.14 The unit serves a wide catchment area in the South-East of England (population >5 million) to support people with complex rehabilitation needs that are beyond the scope of their local rehabilitation services. Set in an acute general hospital, it offers 24 h medical care and caters in particular for people who have ongoing medical problems, and cannot possibly be managed on an outpatient or domiciliary basis. This setting was chosen because it has a high proportion of patients with complex needs.

Sample

Data were collated for a total of 179 consecutive patients (110 males and 69 females) during a 30-month period between 1 June 2006 and 1 December 2008. Of these, 173 had data collected at discharge from the programme (the remaining six were short admissions only). Demographic details are given in table 1.

Measures

Rehabilitation Complexity Scale

The RCS is a 16-point measure with five items10 (see Appendix 1): basic care and support (C: range 0–3); nursing (N: 0–3); therapy (T), the number of therapy disciplines (TD: 0–3) and overall therapy intensity (TI: 0–3); medical (M: 0–3).

Northwick Park Dependency Scale

The NPDS is a measure of nursing needs/interventions specifically developed for rehabilitation settings.5 It is shown to be a valid measure of nursing dependency15 16 and is used increasingly widely both in the UK and abroad.17 18 It has a total score ranging from 0 to 100 and is subdivided into two domains, Basic Care Needs (12 items, score range 0–65) and Special Nursing Needs (seven items, score range 0–35). Scoring levels reflect the number of carers and time taken to complete the task. It is translated by a computerised algorithm into a direct measure of care hours.11

Northwick Park Therapy Dependency Assessment

The NPTDA is an equivalent scale to assess therapy needs/interventions in a neurological rehabilitation setting.7 8 It provides an ordinal score of therapy dependency (range 0–100) consisting of 26 items in two principal domains (direct and indirect intervention). Scoring levels reflect the number of therapists and approximate intervention times. Once again, a computerised algorithm translates this into an estimate of the total therapy hours, subdivided by discipline and by domain.7

Application of measures

In this evaluation, all three measures were applied retrospectively in respect of the levels of intervention provided. This application was chosen for its greater objectivity and precision, and also for comparability with the FIM and BI (see below). They were recorded as part of the unit’s routine practice—RCS, NPDS and NPTDA ratings are rated at fortnightly intervals, based on the average of the preceding 2 week’s interventions for each patient. NPDS scores were rated by their ‘named nurse,’ and NPTDA scores by the treating therapy team. RCS scores were recorded during the multidisciplinary ward round, by the treating team: C and N scales reported by the nurses, and M scale by the doctors. T scale scores were computed from routinely recorded therapy intervention times. RCS, NPTDA and NPDS ratings for start and end of treatment were chosen to represent the period corresponding as closely as possible to the FIM and BI ratings (see below).

Repeatability

The test–retest repeatability was tested during a 7-month period from June 2006 to January 2007.

RCS Care, nursing and medical subscale

C, N and M scores were rated for all current in-patients (usually n=18–20) by the treating clinical team at the beginning of each weekly ward-round meeting. At the end of the meeting (approximately 1.5–2 h later), RCS scores were again rated by the same clinical team, without reference to the first rating. The ward round thus acted as a distracter task. Over the 7-month period, 316 paired ratings were obtained.

RCS Therapy subscale

T scores were extracted from computerised therapy records on this unit. During the same 6-month period, T scores for RCS version 1 based on total therapy hours of intervention per week) were computed in parallel with T scores for RCS version 2 (TD (number of disciplines involved) plus TI (overall therapy intensity) (n=307 paired ratings)).

BI and FIM

The BI is a 10-item measure recorded on a range of 0–20.19 The FIM is an 18-item scale,1 subdivided into a motor scale (13-items, range 7–91) and a cognitive scale (five items, range 5–35). FIM and BI scores were extracted20 from the UK Functional Assessment Measure (UK FIM+FAM)21 which is routinely applied at the start (within 10 days of admission date) and at end (within the last 7 days) of the programme. In accordance with the manual,22 scores reflected the individual’s observed level of dependence, not what they could or might do.

Data collection and clinimetric analyses

Deidentified data were extracted from the unit’s computerised records. Analysis was undertaken using SPSS V.15 (SPSS, Chicago) or STATA V.8, using classical approaches within the framework recommended by Medical Outcomes Trust.23

• Conceptual and measurement model

  The conceptual model for development of the RCS is described briefly above.

• Dimensionality

  The Kaiser–Meyer–Olkin measure of sampling adequacy and Bartlett test of sphericity were used to confirm suitability for factor analysis.24 Dimensionality was examined using exploratory and confirmatory factor analysis. Exploratory factor analysis involved a principal-component analysis with orthogonal (Varimax) rotation. Horn’s method of Parallel Analysis was used as the objective criterion for how many factors to rotate.25 26 Confirmatory factor analysis was conducted on RCS data collected for the same patients, but at discharge (n=173), using the AMOS-16 structural equation software programme within SPSS V.15.

• Reliability

  • Reproducibility (test–retest repeatability) was evaluated for individual items and total scores by testing agreement using quadratic-weighted Cohen κ statistics (STATA V.8).

  • Internal consistency of the RCS was examined using Cronbach’s α and item-total correlations in the reliability analysis module of SPSS V.15.

• Validity

  Construct-related validity was evaluated using one set of ratings per patient at the start of treatment. In the absence of a clear gold standard against which to test criterion validity, concurrent convergent and discriminant validity were assessed through Spearman correlations with the NPDS and NPTDA scores.

• Relationship with other measures

  Correlations with the FIM and BI were also examined to compare their potential to evaluate case complexity in this context.

• Responsiveness was evaluated by testing for significant differences between paired scores recorded at the start and end of the programme (Wilcoxon signed rank test).

Results

Feasibility

In this study, the RCS took less than 1–2 min to administer by a team who was familiar with the scoring manual and used the score regularly in routine practice.

Descriptive statistics

Table 2 shows the descriptive statistics for the RCS, NPDS, NPTDA, FIM and BI scores at the start and end of the programme. As expected for this complex group, RCS scores clustered at the top end of the scale, with a median of 11 (interquartile range 9–12). There was some evidence of a ceiling effect in the RCS scores, when compared with the more detailed Northwick Park Dependency Scales, particularly for the T score (see figure 1).

• Download figure
• Open in new tab
• Download powerpoint

Figure 1

Sunflower plots. Each petal represents one pair of data. Data for both the start and end of the programme are included to ensure representation across the range of the scale. Comparison of the Rehabilitation Complexity Scale (RCS) therapy score with the Northwick Park Therapy Dependency Assessment (NPTDA) and the Northwick Park Dependency Scale (NPDS) shows significant ceiling effects for both the therapy and care/nursing scales, but particularly the former.

Dimensionality

Table 3 shows the results of a principal-components factor analysis on the correlations of the RCS items. All five items loaded ‘moderate’ to ‘high’ on the first unrotated principal component with loadings ranging from 0.52 to 0.79. Only the first two components had eigenvalues >1, together accounting for 68% of the total variance in scores. Parallel analysis indicated a two-factor solution, which was rotated using a Varimax procedure.

• The first factor appears to be ‘Nursing/medical’ care, which accounted for 44% of the variance. The C, N and M items all loaded high (0.65–0.88) on this factor and low (<0.15) on factor 2.

• The second factor appears to be ‘Therapy,’ accounting for 24% of the variance. The two therapy items (TD and TI) both loaded above 0.80 on this factor and low on factor 1.

Confirmatory factor analysis, conducted on RCS scores at discharge (n=173), examined both a one- and a two-factor model. The one-factor model, with all five RCS items loading on a single underlying factor, showed a relatively poor fit with a goodness of fit index=0.95, comparative fit index=0.92, an adjusted goodness of fit index=0.84, and χ²=25.36 (df=5, p=0.000). The two-factor model (factor 1: C, N and M items; factor 2 TD and TI) had an excellent fit with a goodness of fit index=1.0, comparative fit index=1.0, an adjusted goodness of fit index=0.98, and χ²=1.92 (df=4, p=0.750).

Reproducibility

Test–retest repeatability

Weighted κ values for agreement between repeated RCS ratings (n=316 data pairs) were 0.93, 0.96 and 0.94 for the care, nursing and medical subscales respectively, which constitutes ‘almost perfect’ agreement according to the interpretation of Landis and Koch.27 Agreement between T scores for versions 1 and 2 (TD+TI) (n=307 paired ratings) showed ‘substantial’ overall agreement (weighted κ 0.69). Although a small bias towards higher ratings for version 2 reached statistical significance at a numerical level (Wilcoxon z −8.3, p<0.001), this did not affect the median scores (which were five for both versions) nor agreement for the total RCS scores (weighted κ 0.92).

Internal consistency

The coefficient α for the 5-item RCS scale was 0.76. Item-total correlations were all moderate or high: C, 0.75; N, 0.78; TD, 0.51; TI 0.58 and M, 0.65.

Validity and relationship with other measures

In terms of convergent and discriminant validity, we expected to find a close relationship between the RCS care and nursing (C and N) items and the NPDS; and between the two therapy measures (RCS T score and NPTDA) but weaker correlations across the therapy/nursing divide.

Table 4 presents the correlations with the NPDS, NPTDA, BI and FIM scores. A threshold for significance of p<0.01 was adopted to allow for the multiple tests. The RCS total score demonstrated moderately strong correlations with the NPDS and NPTDA (r=0.49 to 0.79, p<0.001). However, within the subscales, there were differential correlations. Only the T scale correlated with the NPTDA (r=0.72), while the NPDS correlated strongly with the C, and N items (0.70 to 0.80) and, to a lesser extent, with the M item (0.38) but only weakly with the T scale (0.26). These relationships suggest that the RCS T score reflects the needs for therapy intervention, and the RCS-C and -N scores reflect care and nursing needs, but that, as expected, these are relatively independent of each other.

Because BI and FIM are measures of independence (as opposed to dependency), negative correlations with the RCS, NPDS and NPTDA were expected, and indeed found. Table 4 demonstrates that, although the RCS showed an overall relationship with these scores (rho−0.47 to −0.72), the FIM and BI were most closely related to the Care and Nursing items. By contrast, the Therapy score showed only modest associations with the BI and FIM Motor scales (rho−0.26), but a stronger correlation with the FIM cognitive scale (rho−0.44), a relationship reflected also in the NPTDA score. Meanwhile the M item showed little or no relationship with either the FIM or BI.

Responsiveness

We did not expect the RCS overall to change markedly during the programme, even though we anticipated that the relative components of care/nursing and therapy might change. Table 1 summarises the changes in RCS, NPDS and NPTDA scores between the start and end of treatment. Overall there was a small, but significant, reduction in RCS—principally reflecting the reduction in care, nursing and medical (C+N+M) needs (Wilcoxon z −9.0 p<0.001), while the therapy component (TD+TI) increased overall (Wilcoxon z −4.6, p<0.001). Figure 2 shows an example of a single case analysis of serial RCS measurements at fortnightly intervals, during a 5-month rehabilitation programme. Care and nursing needs gradually fell during the stay, but therapy needs followed a variable course as the interventions changed at different stages of the programme. Similar patterns were reflected also in the NPTDA/NPDS scores and hours of intervention.

• Download figure
• Open in new tab
• Download powerpoint

Figure 2

Care and nursing interventions showing a gradual reduction over the course of the programme. Therapy interventions are more variable over time. While the Rehabilitation Complexity Scale (RCS) scores are ‘blunter,’ they follow a similar pattern to the more detailed scores and care hours.

Discussion

This evaluation represents a clinimetric analysis of the RCS within the context of a specific group of patients with severe complex neurodisability. Although a full psychometric analysis was beyond the scope of this study, our findings are summarised under the Medical Outcomes Trust framework23 in table 5.

The RCS was feasible to score regularly in routine practice, taking 1–2 min to administer by a team familiar with scoring. C, N and M scores were shown to be highly repeatable; and version 2 (which differs from version 1 only with respect to the T scores) gave equivalent results at a clinical level. The repeatability of the T scores has not been tested in this evaluation.

RCS scores changed differentially over the course of a 3–4-month stay—care, nursing and medical interventions reducing, and therapy scores increasing. While at first sight this might seem surprising, it resonates with clinical experience that, as patients become less dependent on basic nursing and medical care for their survival and health, they become more active in therapies. The observed variability over time was reflected in other measures. This suggests that serial recording may be required to capture changes in rehabilitation needs and the interventions that are provided to meet them, throughout the course of the rehabilitation programme.

We found good evidence for convergent and discriminant validity of the RCS in relation to other measures of nursing and therapy dependency, although there was some potential for bias, as scores were necessarily applied by the same treating team. Ceiling effects (especially in the therapy subscale) underline the need to use more detailed scales to measure complexity at the top end of the scale. Comparison with the FIM (motor) and Barthel Index suggests that these measures of physical disability provide a good indication of needs for care and nursing but are relatively poor indicators of needs for therapy or medical intervention. The closer relationship between the FIM cognitive score and level of therapy intervention is also expected in this group of patients with complex disability. As patients get back on their feet and become relatively independent for basic care activities, residual cognitive problems may still preclude the transition to home, but their needs for cognitive therapies (eg, psychology) may increase. This underlines the importance of including specific indicators of therapy and medical needs in the evaluation of rehabilitation complexity.

The RCS items were originally chosen for their clinical importance as the key determinants of cost of a rehabilitation episode. However, we wished to know whether they could be summed to a total score, as an overall indicator of caseload complexity. Item-total correlations and Cronbach α showed moderate internal consistency, suggesting that the five subscales are broadly cumulative. Nevertheless, exploratory and confirmatory factor analyses both showed strong evidence for two distinct dimensions (‘Nursing/medical care’ and ‘Therapy’). This, together with the differential pattern of change over time, suggests that the Therapy items provide additional information with respect to rehabilitation complexity and should be recorded separately. Therefore, although our previous study10 demonstrated that a total RCS score had some discriminatory value in distinguishing services on the basis of caseload complexity, the findings presented here suggest that summation into two subscales is more appropriate. On the other hand, the four components each have differential impact for staffing requirements and between them provide a profile of rehabilitation needs. Separate reporting of item scores (eg, C2, N1, T3, M1) may therefore be required to facilitate clinical interpretation, in a manner analogous to the Glasgow Coma Scale.

In addition to those already mentioned, the authors recognise a number of limitations to this study:

1. It was confined to a single centre with a particularly complex group of patients undergoing neurological rehabilitation. While it was pertinent to evaluate use of the RCS in this group, because of the anticipated ceiling effects, the majority of patients in this cohort had severe physical disability. Further work is now required to evaluate the RCS as a measure of rehabilitation needs across a broader range of conditions and rehabilitation settings.

2. Our confirmatory factor analysis was undertaken on discharge scores from the same group of patients as the exploratory factor analysis. Even though the two sets of scores were demonstrated to be significantly different, the results must be interpreted with caution, and further confirmatory analysis is required in different patient groups to confirm the factor structure of the RCS.

3. The relationship between rehabilitation needs and intervention is complex, and many clinicians feel frustrated by the lack of resources to meet all their patients' needs for rehabilitation. As noted above, the RCS, NPDS and NPTDA are designed to be applied either prospectively (to predict ‘needs’) or retrospectively (to describe the level of intervention provided) and so ultimately to quantify unmet need. In this study, we used retrospective application to reflect the actual rehabilitation intervention provided, in order to maximise objectivity and precision. Further evaluations are required, employing both prospective and retrospective designs, to establish the relationship between need and intervention, which was not addressed in this study.

In summary, the RCS is a new measure designed to evaluate complexity of rehabilitation needs/intervention. This study has provided evidence for its validity, reliability and responsiveness to change in patients with complex neurological disabilities. It provides a profile of rehabilitation needs, which could be used to distinguish case complexity in rehabilitation, and it includes areas of care that are missing from other routinely used instruments, such as the FIM and BI. Because of ceiling effects, however, caution should be exercised in its application to the most complex cases, where more detailed evaluation may be required using the NPDS and NPTDA.

It is anticipated that some further adjustment or adaptation may be needed to capture the various features that relate to complexity in different clinical settings—for example in neuro-psychiatric units, recording the level of ‘risk’ of harm to self or others may be more relevant than the need for acute medical care. Further work is now warranted to explore the use of the RCS in different rehabilitation settings and with other patient groups.