Objectives: Research concerning the natural history after mild traumatic brain injury (TBI) faces a number of methodological challenges, including those related to subject recruitment. The aim of this study was to determine whether subjects who agree to participate in longitudinal research differ from those who do not. The presence of identifiable, selective factors operating during recruitment may be an important source of systematic bias. In Canada, given the presence of universal healthcare coverage, this issue can be examined using population based, administrative databases to obtain information about a cohort that was approached for study enrolment, regardless of whether they ultimately agreed to participate.
Methods: A sample of 626 consecutive patients with mild TBI was invited to enrol in TBI outcome research. Those who agreed to participate (n=272) were compared with those who refused (n=354) on demographic, past health, and injury related variables. Thereafter, using encrypted health card data, the two groups were contrasted with respect to pre-injury and post-injury healthcare utilisation.
Results: No premorbid differences between the groups emerged. However, all early indices of TBI severity were significantly worse for the participants group (p<0.001). Consistent with these findings, healthcare utilisation rates were no different before injury, but were significantly increased after injury for the participants (p<0.001), even beyond the period of study enrolment (p<0.001). Differences remained even after controlling for those with significant non-TBI injuries.
Conclusions: Premorbid factors did not predict whether patients comply with, or refuse study participation. However, the participants group was biased toward those with more significant injuries, which translated into higher rates of healthcare utilisation after injury. These results strike a cautionary note, given the apparent systematic bias influencing enrolment in longitudinal studies of mild TBI.
- traumatic brain injury
- longitudinal studies
- section bias
- TBI, traumatic brain injury
- GCS, Glasgow Coma Scale
- PTA, post-traumatic amnesia
Statistics from Altmetric.com
Mild traumatic brain injury (TBI) is common,1 with a significant minority of patients developing disabling symptoms of a persistent postconcussional disorder.1–3 Longitudinal studies focusing on outcome, while essential to understanding the natural history of this syndrome, face a particular set of challenges. Foremost among these are difficulties surrounding patient recruitment and the historically high drop out rates.4 Both of these have the potential to confound results and thereby compromise the external validity of study conclusions. For example, Middleboe et al examined the long term effects of mild TBI on general health.5 However, more than 40% of the subjects opted out of serial assessments over a one year follow up period. Compared with study completers, this group was noted to have less prominent symptoms at baseline, suggesting the presence of a selection bias among those completing the study.
In a number of studies, the proportion of subjects lost to follow up has approached 50% or greater.5–10 Some investigators have avoided the problem through the use of restrictive exclusion criteria,11 or by examining a “captive” population such as college athletes.12 However, these approaches have their own limits to generalisability. The potential for selection bias has been examined to only a limited extent within the mild TBI literature. While some authors have ignored the issue, others have varied in their conclusions, suggesting that differences between those who complete or drop out of studies may be negligible9 or substantial.6–8 Although these conclusions are equivocal, what is more certain is the paucity of comparable data looking at who agrees to enrol in the first place, at the point of study inception.
For obvious reasons, it may be virtually impossible to obtain detailed information for those who decline study participation. Information contained in medical notes is limited and at times inaccurate, particularly with regard to pre-injury status. Yet these data are of importance for two reasons. Firstly, the presence of identifiable, selective factors operating during patient recruitment may be a source of systematic bias for longitudinal research. Secondly, premorbid factors themselves have been suggested to play a key part in the persistence, if not development, of postconcussional sequelae.13,14
A means to circumvent these constraints is offered by the administrative databases associated with the Canadian health insurance plan, a system that provides universal health services to all residents. Analyses based upon individual health insurance numbers can be undertaken, while preserving patient anonymity, to examine medical service use both pre-injury and post-injury. When combined with demographic and injury related data, this approach offers a unique look at a cohort who was approached for study enrolment, regardless of whether they ultimately agreed to participate.
All patients with mild TBI attending a regional trauma centre in Ontario, Canada between January 1998 and June 2000 were invited to participate in TBI outcome research. Patients included those presenting directly to the facility’s emergency department, as well as those referred from other hospitals with limited trauma facilities. Research assistants monitored the emergency and trauma wards on a daily basis to identify new admissions, who were approached within days of injury. They were contacted either by telephone, or directly, if discharge had not yet occurred. The interviewer identified herself as affiliated with the hospital TBI service. Subjects were invited to enrol in a research study that would entail a 90 minute interview to complete a number of questionnaires about postconcussional sequelae and a limited neurocognitive assessment. The study was to include reassessment at six months after injury. Financial incentive was limited to meal and parking expenses. The sample for the study comprised 626 consecutive patients.
Mild TBI was defined according to the criteria proposed by the American College of Rehabilitation Medicine: loss (or altered level) of consciousness for not more than 20 minutes; duration of post-traumatic amnesia (PTA) <24 hours; and Glasgow Coma Scale (GCS) score ≥13,15 accompanying a history of trauma or acceleration-deceleration movement applied to the head. Subjects were aged 15–65 years. Patients were divided into those who agreed to participate in TBI research (the “participants” group) and those who declined enrolment (the “refusals” group).
Subject information was gathered in two stages. Firstly, demographic, past health, and injury related data were collected via patient interview and examination of the hospital record. These included age, sex, marital and employment status, and years of education; past history of head injury, health difficulties, and substance misuse. Mechanism of injury and severity indices such as GCS, duration of post-traumatic amnesia, and the presence or absence of abnormalities on brain computed tomography were also noted. By convention, GCS recorded on arrival at the emergency room was used.6 Duration of PTA was estimated as early as possible using the method employed by Russell and Smith.16–18
Thereafter, information concerning each subject (participants and refusals) were extracted from two large, administrative databases.
Ontario Health Insurance Plan. Physician services provided to Canadian residents are funded by health insurance plans within each province. Service information is maintained in a database, the content of which is comparatively accurate as information submitted by physicians is associated with a reimbursement fee. Through the Institute for Clinical Evaluative Sciences (ICES), an independent research facility established by the Ontario Ministry of Health,19 this information can be accessed for analysis. Confidentiality is preserved, as patient names are removed and the unique health card numbers encrypted before accessing the database. Analyses are then reported using aggregate statistics. As a consequence, information cannot be associated with a specific individual, in keeping with confidentiality protocols.
Medical claims for both the participants and refusals were examined. Services rendered included inpatient/outpatient consultations, minor assessments, diagnostic procedures, and therapeutic interventions. These were combined to yield the mean number of medical services utilised by members of each group during specified time periods. As such, “services” is not synonymous with specific doctor “visits”, as more than one service may be provided and submitted to the healthcare plan within a visit. An adjusted “per year” rate of medical service utilisation was calculated for the following: (a) an approximately eight year period before injury; (b) the first six months after injury; and (c) a 15 month period subsequent to (b), when formal participation in studies had ceased.
Statistics Canada 1996 Population Census.20 This permitted estimation of family income, utilising each subject’s home postal code. Separate postal codes are assigned to small subdivisions within a neighbourhood—5–10 addresses on a street, for example. From the census database, estimates of median family income have been derived for households sharing the same postal code. The analyses were again performed within ICES using aggregate statistics to maintain patient anonymity.
The project was approved by the research ethics committee at the University of Toronto. Furthermore, access to the administrative databases conforms with confidentiality protocols within ICES and the Ontario Ministry of Health.19
Demographic, past health, and injury related data were analysed using a statistical package for personal computer.21 Administrative database information was analysed within ICES. Between groups comparisons were undertaken using parametic and non-parametric tests, where appropriate. A significance level of 0.05% was used.
Of the 626 persons comprising the sample, 272 agreed to participate in research (43.5%); the remainder declined. The mean age for the entire sample was 32.6 (SD 12.2) years; 66.8% were male. Valid provincial health insurance numbers could be obtained for 598 of the 626 (96.6% of the participants and 94.3% of the refusals respectively). Valid postal codes were available for all (n=626).
For those who declined study enrolment, certain data proved to be largely unobtainable because of limited contact. Several variables were therefore dropped from the analysis, including years of education, occupation, and history of past TBI, psychiatric illness, or substance misuse.
A summary of demographic and pre-injury health data for the full cohort is presented in table 1. No significant differences emerged with respect to age, sex, marital status, or the proportion employed full time at the time of injury. Full time students were included among those employed full time. The number of students in each group was similar (χ2=1.24; p=0.305). The rate of past important medical illness did not differ between the groups.
Injury related data for the full cohort are also shown in table 1. The two groups showed a consistent difference with respect to the early indices of TBI severity. Participants were significantly worse with respect to initial loss of consciousness, GCS scores in the emergency room, duration of PTA, and the presence of early CT abnormalities. The latter was defined as intracerebral contusion/haematoma or subdural/epidural haemorrhage. A greater percentage was scanned, again reflecting the presumption of greater TBI severity by the trauma physicians. Participants also had a higher rate of injuries to other systems, defined as significant orthopaedic and/or internal (thoracic/abdominal) injuries. Consistent with these differences, participants were more frequently admitted to hospital for treatment, rather than released directly from the emergency room. Finally, mechanism of injury differed between the groups. Subjects in the participants group were more frequently involved in motor vehicle accidents as compared with all other types of injuries combined. The latter included sports related injuries, falls, assaults, and industrial accidents.
To exclude the effects of other systems injuries on study participation, those with TBI alone were examined in a subsidiary analysis involving 394 subjects. These data are shown in table 2. Again, there were no significant differences with respect to age, sex, marital status, proportion employed full time, or past history of important medical illness. Mechanism of injury was no longer different. Thus, while motor vehicle accident related trauma was associated with the presence of injuries to other systems, it did not seem to predict study participation in and of itself. Although GCS scores were similar, the groups continued to differ with respect to PTA duration and CT results, again indicating that study participants tended to have worse injuries, regardless of additional orthopaedic or internal injuries.
Results of the administrative database analyses are presented in table 3. There was no difference in the pre-injury rate of medical service use between the groups. This clearly changed after injury, both for the first six months after the accident, as well as the subsequent 15 months, in which participants had higher rates of service use. These differences held when subjects with other systems injuries were excluded (table 4).
The Statistics Canada census analysis indicated a lower median family income for the participants group when the entire cohort was examined (table 3). This difference was not large in actual amount. Furthermore, the discrepancy disappeared when the effects of significant non-TBI injuries were excluded (table 4).
This work represents a novel contribution to the literature regarding the acquisition of longitudinal data in mild TBI research. Utilising a provincial health insurance database along with hospital records, it was possible to obtain both pre-injury and post-injury information about a group of patients who declined study enrolment—data that would have been otherwise inaccessible. In addition, pre-injury data were free from the effects of recall bias, an influence that can be substantial among those who have experienced a significant trauma.22
Our first observation was of similarity between participant and refusal groups before injury, including demographics, socioeconomic status, and pre-injury health status/service utilisation. Thus patients agreeable to enrol in research did not appear to represent a group that was predisposed to more frequent involvement with physicians or other medical services. Because the administrative database captures every medical service rendered, the absence of between group differences also tends to rule against significant pre-injury psychological differences, based upon research linking rates of doctor visits and the presence of psychological distress or psychiatric syndromes.23–25
The second principal finding was that despite premorbid similarities, study participants had more significant head injuries, plus more frequent injury to other systems, than those who refused enrolment. Differences in PTA duration and CT results remained even after adjustment for other injuries. In turn, this disparity translated into higher rates of medical service utilisation after injury for the participants group—also independent from other systems injuries—that persisted beyond the period of study involvement.
While the presence of between groups differences does not invariably introduce bias, discrepancies among characteristics known to mediate outcome are more concerning. Specifically, GCS, PTA duration, the presence of CT abnormalities, as well as injuries to other systems have all been associated with variability in outcome after mild TBI.11,26,27 Our results thus raise the possibility of systematic bias in mild TBI outcome research from the point of enrolment onward. In contrast, the results do not support selective participation based upon premorbid differences. This too is relevant, because age,13 educational level,29 employment status,30 as well as premorbid psychological adjustment and psychiatric health13,14 are also thought to influence outcome after mild TBI.
To our knowledge, this is the first report to formally investigate the question of recruitment or participation bias within the TBI population. Although appreciated for some time within the general medical literature, the paucity of studies reporting the rates and characteristics of those excluded from studies has been noted.31–35 This holds true for the mild TBI literature, where samples generally comprise “willing” participants. Those who refuse enrolment are not discussed.10,11,36 Only the report of Bohnen et al8 describes subjects who declined enrolment at inception: of 131 consecutive patients, 71 (54%) “were willing to participate” in a study of postconcussional symptoms after mild TBI.8 The authors comment that a portion of the refusals were younger, intoxicated men; however statistical analysis was not provided.
Although this result was similar to ours, the applicability of our findings to mild TBI research in general remains to be clarified. It may be that other centres obtain higher participation rates at the point of inception; if so, however, this has not been reported. Our methods of recruitment are straightforward—that is, using a research assistant to approach consecutive patients admitted through an emergency room—and common to many studies.10,37 The sample population did not differ from that of other tertiary trauma centres; nor did the study entail a particularly onerous protocol or time commitment. As such, it is probable that other investigators would encounter similar recruitment issues in the study of MHI. Until specifically dealt with in their methodology, the suspicion of ascertainment bias remains.
Difficulties with subject attrition over the course of follow up have been noted in the introduction. Although different from recruitment bias, these issues are related as both depend upon subject compliance. In a meta-analytical review of mild TBI outcome studies, Binder38 points to evidence for a selection bias operating over time—namely, that those who persist with studies may be more ill. A similar bias may influence those who consent to participation in the first place: those with milder injuries may have already recovered, and thus see no reason to participate. Commitments such as work or school may render them less available for study involvement.
Our results strike a cautionary note for mild TBI research. It is known that conclusions regarding natural history may be distorted whenever a sample is not drawn from among all patients within a population. By considering subjects admitted via an emergency room or trauma service only, those with more severe or complex cases may be over-represented, and outcome may seem worse.39 Furthermore, the observed rates of comorbid conditions may be higher, given the greater likelihood that individuals with two (or more) disorders seek help than those with only one.40 Our results suggest that these factors will be aggravated if the issue of selective enrolment is not dealt with. Thus current estimates of poor outcome in mild TBI, variably placed at 10%–20%,1 may be overstated. Similarly, characteristics “associated” with mild TBI may also be overestimated, including putative “risk factors” for poor outcome. This may in turn distort perceptions of causation. It is therefore recommended that future research examining the mild TBI population investigate the potential impact of recruitment bias on study results.
Donald Redelmeier’s supervision of the ICES analyses and Katy Kamkar’s meticulous research assistance are gratefully acknowledged. This work was financially supported by the Physician’s Services Incorporated Foundation.
Competing interests: none declared.