Article Text
Abstract
Background Key performance indicators (KPIs) are metrics that compare actual care against an ideal structure, process or outcome standard. KPIs designed to assess performance in deployed military surgical facilities have previously been published. This study aimed to review the overall performance of surgical trauma care for casualties treated at Role 3 Camp Bastion, Medical Treatment Facility, Afghanistan, in light of the existing Defence Medical Services (DMS) KPIs. The secondary aims were to assess the utility of the surgical KPIs and make recommendations for future surgical trauma care review.
Methods Data on 22 surgical parameters were prospectively collected for 150 injured patients who had primary surgery at Camp Bastion between 1 May 2013 and 20 August 2013. Additional information for these patients was obtained using the Joint Theatre Trauma Register. The authors assessed data recording, applicability and compliance with the KPIs.
Results Median data recording was 100% (IQR 98%–100%), median applicability was 56% (IQR 10%–99%) and median compliance was 78% (IQR 58%–93%). One KPI was not applicable to any patient in our population. Eleven KPIs achieved >80% compliance, five KPIs had 80%–60% compliance and five KPIs had <60% compliance. Recommendations are made for minor modifications to the current KPIs.
Conclusion 78% compliance with the DMS KPIs provides a snapshot of the performance of the surgical aspect of military trauma care in 2013. The KPIs highlight areas for improvement in service delivery. Individual KPI development should be driven by evidence and reflect advances in practice and knowledge. A method of stakeholder consultation, and sequential refinement following evidence review, may be the right process to develop the future set of DMS KPIs.
- SURGERY
- AUDIT
- TRAUMA MANAGEMENT
Statistics from Altmetric.com
Key messages
Key performance indicators designed to assess performance in deployed military surgical facilities were published in 2008.
Overall compliance with key performance indicators between May and August 2013 in Camp Bastion was 78%.
From this analysis of performance, areas for improvement in service delivery are highlighted.
Development of the individual performance indicators themselves should be driven by evidence and reflect advances in practice and knowledge.
Further review is suggested to revise the key performance indicators within the military trauma system.
Introduction
Key performance indicators (KPIs) are measurements that compare actual care against an ideal structure, process or outcome standard.1 Increasingly used to obtain objective data for assessing quality and implementing performance improvements, they are considered by many to be fundamental to the provision of excellent care.2–4 Despite the nearly universal adoption of KPIs,5–7 there is little consensus about the process of KPI generation;8 further doubts remain on their accuracy for identifying quality of care problems.9 ,10
During the Afghanistan and Iraq conflicts, the UK Defence Medical Services (DMS) have provided surgical care within a wider military trauma system. In 2008, authors from the British Military sought to develop KPIs, which reflected both the modern military setting and, where possible, the best of contemporary supporting evidence; a set of 60 bespoke KPIs for the end-to-end trauma care was published.11 These KPIs form an established component of the cyclical process of continuous quality improvement known as the ‘Major Trauma Audit for Clinical Effectiveness’ within the DMS.12
The primary aim of this project was to review the overall performance of the surgical aspects of trauma care in a Role 3 Medical Treatment Facility (MTF) using the surgical KPIs. The secondary aims were to assess the utility of the surgical KPIs and make recommendations for future surgical trauma care review. Data were collected from 150 consecutive trauma patients undergoing surgery at the Role 3 MTF in Camp Bastion, Afghanistan, and their treatment examined against KPI standards.
Materials and methods
Study setting
The study was approved and registered with the Medical Directorate, Joint Medical Command, Royal Centre of Defence Medicine. Patients underwent surgery at the Joint Force Medical Group, Role 3, MTF in Camp Bastion, Helmand Province, Afghanistan. The MTF was a UK-led multinational hospital, which provided all care to combatants and life, limb and eyesight saving treatment to civilians, in accordance with the Geneva Conventions.13 ,14 Role 3 care provides advanced intheatre subspeciality patient care, equivalent to most civilian American College of Surgeons-Committee on Trauma Level-II trauma centres.15
Key performance indicators
Due to the evolution of the Trauma system, the original 2008 ‘Operating Theatre’ KPIs11 were modified. At the time this study was registered with the Medical Directorate, an internal review within the Academic Department of Military Surgery and Trauma (ADMST), Royal Centre for Defence Medicine, examined which significant developments within the damage control pathway had been adopted since the KPIs were first published. These were related to the adoption of damage control resuscitation and included the use of tranexamic acid, thromboelastographic (ROTEM) analysis to diagnose coagulopathy and blood gas analysis to monitor the degree of acidosis; a modified WHO surgical checklist had also been mandated and was therefore included as additional KPIs. Of the original 19 KPIs, one was removed, two modified and four added to give a total of 22 parameters in this study (Table 1).
Surgical care
Acute surgery is a small but significant component of the total care of a casualty. Regardless of casualty affiliation, all surgery in this study was conducted in one location. Missing data due to variable casualty pathways preoperatively were minimised by measuring only the surgical KPIs.
Data collection
Data were prospectively collected for 150 patients between 1 May 2013 and 20 August 2013. Adult and paediatric patients who sustained a traumatic injury and underwent initial surgery at the Role 3 MTF, Camp Bastion, were eligible for inclusion. Patients were excluded if they were transferred from another Role 2 or 3 facility with surgical capability or if they did not sustain traumatic injuries. Only the first surgical episode for each patient was recorded, and episodes were excluded from analysis if <70% of the data points recorded.
The 22 KPI outcomes were recorded for each casualty by a registered nurse employed in the operating department (CLH; Table 3). For each KPI, the possible outcomes were ‘Yes’ if the KPI was complied with, ‘No’ if the KPI was not complied with and ‘N/A’ if the KPI was not applicable for that patient. An individual KPI was deemed not applicable for a patient if the patient did not have an injury that the KPI measured—for example, head injury parameters in a patient with isolated lower limb injuries. The applicability of each individual KPI was judged according to injuries sustained and haemodynamic status at the time of data recording (CLH). Subsequently, applicability was reviewed again by two authors, both surgical trainees (AES/MERM), by comparison of injury data held in the Joint Theatre Trauma Registry (JTTR) and compared with the original data collection performed by CLH. Additional data to provide context for interpretation were collected using the JTTR and included patient demographics, mechanism of injury and treatment timelines.
Data analysis
Each individual KPI was analysed in three separate domains:
Data recording of the KPI: this included all valid data entries for a specific KPI, that is, Yes AND No AND N/A entries.
% of KPI data recorded=100 × (number of valid entries)/(total sample population).
Applicability of the KPI: this included all relevant results, that is, Yes AND No entries.
% of recorded KPI data that is applicable = 100 × (number of applicable entries)/(number of yes OR no OR N/A entries).
Compliance with the KPI
% of KPI that are compliant = 100 × (number of yes entries)/(number of yes AND no entries).
The results of each of these KPI domains were ‘RAG coded’ (red, amber or green) comparable with KPI coding within the NHS.16 ,17 Cut-offs were assigned arbitrarily such that red (<60%), amber (80%–60%) and green (>80%). These values were displayed graphically, and KPIs were numbered for convenience of communication.
Results
Eleven of the 150 patients sampled were excluded from analysis (Figure 1), leaving 139 eligible patients for data analysis (Table 2).
Table 3 displays the results for each KPI. Overall median data recording was 99.6% (IQR 97.8%–100%). Data were recorded poorly for KPI 17, ‘Were all wounds photographed pre and post debridement and copies available in the UK?’ Overall median KPI applicability was 55.8% (IQR 10.1%–99.1%). KPI 22, ‘Were decompressive craniotomy/craniectomy performed <4 hours of a blunt head injury?’ was not applicable to any patient in our sample. It was therefore not included in subsequent summary compliance calculations. Of the remaining 21 KPIs, median KPI compliance was 77.9% (IQR 57.6%–93.1%); 11 KPIs were scored green for compliance (>80%), 5 amber (80%–60%) and 5 red (<60%).
Discussion
The primary aim of this study was to review the performance of surgical trauma care in war-wounded casualties through compliance with the 2008 ‘operating theatre’ KPIs.11 The overall compliance with 21 surgical KPIs at Camp Bastion between 1 May 2013 and 20 August 2013 was 78%. The study's secondary aim was to address the KPIs themselves and attempt to improve the surgical KPIs for future use. KPI utility was assessed by applicability and clarity. Applicability of all the KPI to the cohort of patients was 56% and we report data recording of 100%. The data recording percentage was intended to act as a surrogate for clarity; subsequently, it has become apparent that it is possible to collect data and not have a clear KPI. Hence, although we record data at 100%, the authors believe that minor modifications to the wording of the KPI will be beneficial to ensure accurate capture of the intended performance indicator; improving KPI specificity (Table 4).
Performance of surgical trauma care (compliance)
The purpose of trauma audit is to identify unexpected outcomes and to use the results to determine why the system's overall performance is above or below expectations,19 in order to drive sustained improvements of care for our patients. Looking at the compliance values of the KPI provides a point in time assessment of the performance of surgical trauma care, as judged by indicators that Stannard et al deemed appropriate in 2008. A root cause analysis of the amber and red KPIs was performed.
KPIs 12 and 13 relate to the use of the WHO checklist and recording of preoperative temperature; there appears to be an adequate quantity of data collected for these and they are unambiguous. KPI 19 relates to the documentation of postoperative care plans including a nutrition element—with a minor modification (Table 4) this KPI is also clear. The lack of compliance for this KPI was predominately due to poor documentation of a nutrition plan. Overall, for KPIs 12, 13 and 19, the amber and red scores likely reflect the performance of the system and may represent areas for improvement in crew resource management service delivery.20
KPI 14 concerns the use of tranexamic acid (TXA). Interpretation of the compliance of this KPI is complicated by virtue of the drug potentially being administered in multiple locations from the point of wounding to the operating theatre. The data capture in this study was performed in the Role 3 only—it is conceivable that some patients were given TXA in the pre-hospital phase and our study did not identify these patients. We recommend that this KPI is moved to the Emergency Department bundle.
KPI 15 measures whether fasciotomies were performed for confirmed vascular injuries. This is relatively difficult to assess prospectively by anyone other than those directly involved in the operation itself. In our study, data collection was performed by a theatre nurse and it is conceivable that applicability of this KPI was not accurately assessed. We suggest a change in the wording of the KPI (Table 4) to reflect that arterial injuries may need a fasciotomy and propose that in future the operating surgeons document this specific KPI.
KPIs 16 and 18 address casualties with ‘shock’ being taken to the operating theatre in <30 min—they were scored as amber and red, respectively. Interpretation of these results maybe more complex than the simple colour coding suggests. Modern purpose built trauma care facilities present environments where resuscitation can be continued while imaging is performed. In casualties with active haemorrhage, harm-to-benefit assessments are made between imaging and transfer directly to theatre for definitive control of haemorrhage. Smith et al21 reported that CT delayed time to theatre by 41 min compared with casualties who did not have a CT prior to surgery, but in abdominal injury, the use of CT is correlated with a reduction in non-therapeutic laparotomy.22 As a desire for increased preoperative information seems to represent the decision-making preference of military surgeons during this study period, it is reasonable to expect a low compliance rate with a 30 min cut-off for these KPIs. Additionally, the term ‘shock’ is poorly defined and applies in a wide range of haemodynamic states. Adequate interpretation of these results needs patient outcome data, which this study did not encompass. Other studies have begun to support this practice.23
KPI 20, ‘Were all wounds ICRC scored at initial surgery?’ and KPI 21, ‘Were bacteriologic specimens taken pre and post debridement with results available to clinicians?’ were not complied with. We believe that this is because clinicians in the MTF did not believe that these measures had an impact on patient outcome. Therefore, resources were not spent on fulfilling these tasks. As this study does not address outcome, we are unable to make recommendations to support or refute following these KPIs.
Utility of KPIs
An intention of the KPIs is to provide data on the performance of the system and act as a driver for performance improvement. In the future, KPIs may also be used for accountability and research, but they need further scientific evaluation before these additional roles are introduced.10
By assessing the utility of each KPI, a refined set of KPI was developed (Figure 2). Several had low applicability values in our study such as KPI 22, ‘Were decompressive craniotomy/craniectomy performed <4 hours of a blunt head injury?’ During the study period, the MTF at Camp Bastion did not have a dedicated neurosurgical capability; patients known to have required neurosurgical intervention at the prehospital phase were diverted to a coalition unit at a different site with a dedicated neurosurgical capability. As a result of this policy, patients with head injury have been selected out of this study population. The applicability score of 0% reflects the complex system of trauma care delivered in this mature environment. This example raises two important points: Ragel et al24 demonstrated that surgical management of head injury remains a relevant topic in military neurosurgery and so the British Military's KPIs must continue to reflect this. The second point is about the context of trauma care in a mature environment; while it is important to focus on specific aspects of patient care, the interlinked trauma system from the point of wounding to rehabilitation is key to a casualty's outcome. It is now beyond reasonable doubt that the performance of a trauma system depends on many steps in the chain of casualty care. It is the trauma system as a whole that impacts on casualty outcome.
Modification of KPIs in the future
This study is the first that reports compliance with surgical KPIs in military trauma. As reflections on lessons learnt from recent conflicts continue, it is an opportune moment to review the military KPIs in their entirety. They must evolve to reflect new evidence, techniques and changes in the processes of care; just as the American College of Surgeons publish revisions to their quality indicators,25 so too will the DMS indicators need revising. Whether regular or ad hoc review is required is not clear. The ‘Operating Theatre’ KPIs11 were developed for OP HERRICK in 2007 and were intended to be bespoke for that military and surgical environment. In 2015, the British Military no longer operates in a mature conflict. Future military engagement will involve reconfiguration of the trauma system. The future DMS KPIs either need to reflect the contemporaneous operating environment (whether that is humanitarian, conventional state-on-state or asymmetric conflicts) or generate generic KPIs that are applicable across all operational environments. If generic evidence-based KPIs are desired then following the example of KPIs selection by Santana et al appears to be the optimal solution.
The Santana group undertook a systematic review which listed 1572 indicators in use globally. Through a process of stakeholder consultation and sequential refinement, 31 indicators with a substantial evidence base and end user support (eg, clinicians) were endorsed. Quality indicators were measured in nine dimensions (Box 1). Such a systematic review aims to ensure KPIs continue to assess safety, effectiveness, efficiency, timeliness, equity and patient centredness through structures, processes and outcomes.10 Their methodology should be considered as the gold standard in future refinement of the DMS KPI.
Santana's nine dimensions to measure quality indicators10
▸ Targets important improvements
▸ Precisely defined
▸ Reliable
▸ Valid
▸ Can be implemented with risk adjustment
▸ Can be implemented with reasonable cost
▸ Data collection effort is low
▸ Results easily interpreted
▸ Globally is a good indicator (overall evaluation)
Limitations
The methodology of this study did not permit analysis of whether compliance with the KPIs influenced patient outcome. This study was, therefore, not able to propose a significantly different set of surgical KPIs due to the lack of evidence on outcome.
Conclusion
This study assessed a published system of KPIs in the dynamic and pressured environment of the combat surgical arena, and compliance was 78%. It appears that compliance with KPIs could be improved in some areas, which may lead to improvement in overall care.
Useful data from KPIs are only possible once they have been rigorously assessed as being applicable to the patient group, conducive for data collection and influence patient outcome. This study represents a further step in the direction of ongoing KPI review, and a further more inclusive review of the British Military's KPIs is recommended to continue this progress. A method of stakeholder consultation, and sequential refinement following evidence review, may be the right process to develop the future set of DMS KPIs.
Acknowledgments
We thank Mr G Terry, Head of Planning and Performance, Great Ormond Street Hospital for Children NHS trust, Great Ormond Street, London, for his advice regarding root cause analysis and RAG coding. We thank Majors R Faulconer and J Berry, Royal Army Medical Corps, for their assistance with enabling the study. We also thank the Clinical Information and Exploitation Team, Joint Medical Command, and UK Defence Statistics (Health) for collecting, collating and identifying appropriate data for this article.
References
Footnotes
Twitter Follow Max Marsden at @maxmarsden83
Contributors CLH collected the data, MERM and AES contributed equally to the data analysis and preparation of the manuscript, NJN and MJM conceived and developed the study and revised the manuscript. DMB also revised the manuscript. MERM is responsible for the overall content as guarantor.
Competing interests None declared.
Ethics approval Medical Directorate, Joint Medical Command, Royal Centre of Defence Medicine.
Provenance and peer review Not commissioned; externally peer reviewed.