Background: To examine nurse-physician inter-observer agreement on triage categorization and analyze their differences for future reference.
Methods: A retrospective observational study was performed. Patients entering a 3500-bed medical center emergency department (ED) from July 1 to 31, 1998 were randomly selected. We compared triage assignments made by nurses and 2 ED physicians, and examined them for inter-observer agreement (kappa-statistic) within each illness category.
Results: We found that the overall nurse-physician agreement on triage categorization had a κ-value of 0.32 (99% confidence interval, 0.27-0.37). The level of inter-observer agreement was not consistent across all illness categories. Agreement was better when assigning critical patients, but it was poor when assigning non-emergency patients.
Conclusion: The overall nurse-physician agreement with triage categorization was poor. The lack of agreement on triage decision making has important implications for EDs in which the priority of care is based on nursing triage categorization. Detailed chart recording and continued work is necessary to improve the agreement between nurse-physician triage categorization.
(Chang Gung Med J 2002;25:446-52)

Key words: triage, nurse-physician inter-observer agreement, emergency department.

Triage is a critical component of any emergency department (ED). The main goal of a triage system is to assign a degree of urgency to each patient, so that the most seriously injured or ill receive medical attention more quickly. Triage is also used to measure the ED workload and to predict resource utilization. Despite its importance and widespread use, there is little agreement on triage protocols and poor validation of its relationship to important outcome variables.
Existing triage systems use from 2 to 7 priority categories. Degree of urgency is decided based on patient complaints, physiological measurements, premorbid illness, test results, and historical cues concerning the current illness.(1,2) Nurses who have received training usually perform triage categorization. The benefits claimed for triage include improved patient flow resulting in a reduction in waiting times, and increased patient satisfaction.(3)
Several studies(2-21) have been conducted to evaluate different triage protocols, and most of these studies showed that nurse-physician inter-observer agreement with triage levels is poor to fair.(3-6,22) Very few authors have studied this relationship to departmental performance, patient satisfaction, and clinical outcomes. Some studies reported that triage systems do improve patient flow in emergency departments;(2,10) however, 1 study suggested that the triage system was related to increased patient waiting time and unaffected or decreased patient satisfaction.(21) A triage system's ability to predict clinical outcomes has also not been established. One study showed that none of the triage decisions made by nurses, physicians, or a computer was able to accurately predict hospitalization.(5)
In Taiwan, a new triage protocol has been developed in response to implementation of the National Health Insurance Plan, and has been used in most emergency department since May 1998. Using the triage criteria, triage nurses assign a priority to a patient to be seen and treated according to the triage system. No studies have yet investigated the reliability and validity of the triage protocol. The purpose of this study was to examine nurse-physician inter-observer agreement on triage levels using this triage protocol, and to further determine if the level of agreement is consistent among different illness categories.

METHODS

We conducted a retrospective observational study in a 3500-bed tertiary medical center ED. Using the ED database of all patients who visited our ED from July 1 to 31, 1998, every third patient on the list was selected until a sample size of 2200 subjects was reached. Patients who were 17 years of age or younger were excluded from the study due to the different emergency setting and triage criteria for pediatric patients.

The triage categories were defined as follows
I. Patients assigned to triage level I require immediate medical attention, or their lives could be in danger. Examples of such conditions are cardiac or respiratory arrest, chest pain of cardiac origin, suspected myocardial infarction, internal or large external bleeding, seizures, sudden loss of consciousness, open fractures, rape, respiratory distress, systolic blood pressure (BP) < 90 or > 220 mmHg, and body temperature (BT) > 41 or < 32oC.
II. In level II, patients should receive medical attention within 10 min. The delay poses no immediate threat to life or limb, but the patient is in severe pain, or the vital signs are abnormal. Examples of these conditions are syncope, sudden onset of neurological deficit, severe pain, chest pain without definite etiology, moderate asthma, suicidal tendency, violent behavior against others, a small open wound, and possible fracture. Other indications include systolic BP between 180 and 220 mmHg, and BT between 39 and 40oC or between 32 and 35oC.
III. Assignment to triage level III means that ED care is needed, and the patient should receive medical attention within 30 min. Examples of these conditions are fever greater than 39oC, mild injuries without an open wound, headaches, stomach upset, diarrhea, dizziness, black stool passage, back strain, and skin rash.
IV. Assignment to triage level IV indicates that the condition with which the patient presents is non-urgent, that medical care can be delayed, and that the patient should be referred to appropriate alternatives.
For each subject, we recorded the chart number, gender, age, date and time of ED presentation, triage category assigned by the nurses, date and time of leaving the ED, chief complaint(s), vital signs, final diagnosis, and disposition from the ED.
In addition to the triage assignment made by nurses, 2 emergency physicians, 1 attending and 1 senior resident, retrospectively assessed patient conditions from the triage record and chart, and then assigned each patient a triage category using the same triage protocol. The triage note, developed for each patient, included information on age, vital signs, coma scale, chief complaint(s), and brief past and current medical histories. The 2 physicians initially assessed and assigned a triage category to each patient independently, and then they discussed the cases for which they had assigned different triage categories for the same patient and tried to reach a consensus. During the review process, the 2 physicians also discussed the conditions which were not listed in the triage protocol, and then agreed to the assignment of the triage category. The research assistant then added the decisions to the triage protocol, which provided future references for chart review and triage assignments.
To evaluate nurse-physician agreement on triage categories, the chance-adjusted measure of agreement (κ-value) was calculated. Although there is no universal agreement on κ-values that represent "good" or "poor" agreement, several authors have suggested that a κ-value of 0.5 is often consider fair agreement.(4,5) Kappa statistics were compared by calculating the standard error.

RESULTS

A sample of 2200 patients was initially selected from the ED network using a systematic sampling method, and 38 patients were subsequently excluded due to incomplete records in their ED charts. Of the 2162 patients enrolled in the study, 59% were men and 41% were women. The mean age of the study subjects was 48.6 years, with 75% aged between 18 and 64 years, and 25% 65 years and older. Nineteen percent of patients were trauma cases and 81% were non-trauma cases. The total number and percentage of patients in each triage level assigned by nurses and physicians are shown in Table 1.
Table 2 compares the overall agreement on triage categories as determined by nurses and the 2 ED physicians. There was poor agreement between the 2 assignments (Cohen's kappa=0.32; 99% confidence interval [CI], 0.27-0.37), and it is clear that the physicians tended to assign higher degrees of urgency to patients than did the nurses. The nurse-physician agreement on triage categories was poor for both trauma and non-trauma cases, with κ-values of 0.31 and 0.28, respectively. Table 3 shows the list of categories of presenting conditions and the kappa values within each category. The level of physician-nurse agreement was not consistent across all illness groups. Within the non-trauma group, physician-nurse agreement was poor across all illness conditions, with higher κ-values for joint/ skeletal and infectious illness categories. Within the trauma group, agreement as indicated by κ-value exceeded 0.4 for cases with burns and head/spinal injuries, and was complete for cases with other conditions (including dead on arrival, injuries to internal organs, and suicide).

DISCUSSION

This study demonstrates that physicians and nurses had poor agreement on triage assignments. There are possible explanations for our findings. Limitations of our retrospective study might have biased our results. Medical charts do not provide the wealth of information that a physician can obtain from seeing a patient, and possible inadequate recording on the medical charts could also have biased our results. However, another study demonstrated that physicians who saw patients at triage were not likely to agree with nurse triage categorization.(4) This is because physicians' visual assessment added 2 elements to the triage process. First, the physician was able to ask the patient questions he or she considered most important. The second advantage was that a patient's physical appearance may greatly influence a physician's decision making.
One study reported that the agreement between nurses' prospective triage categorization and physicians' retrospective triage categorization was poor (κ=0.18).(6) Brillman et al. found fair agreement between nurses' and physicians' prospective triage decisions (κ=0.45).(5) Another study showed that the agreement between 2 clinicians was fair (κ=0.42).(3)
This study shows that the level of physician-nurse inter-observer agreement was not uniform across all diagnostic groups. We found that physician-nurse inter-observer agreement was better for many trauma cases and a few non-trauma cases (involving the joint/skeleton and infection); this finding is supported by other studies. It is believed that non-trauma cases may be more difficult to assess, and therefore, would result in more discrepant assignments; on the contrary, trauma cases may be more straightforward and are easier to assess quickly by nurses and physicians.
This study also demonstrates that physicians tended to assign higher degrees of urgency to patients, regardless of their diagnostic groups, than did nurses. This could be due to the fact that the physicians we chose were more conservative, especially since they did not actually see and interview the patients. For example, the physician would probably place an elderly patient complaining of chest pain into category 1 if the chart record did not give detailed information on whether the pain was of cardiac or noncardiac origin. For patients presenting with loss of consciousness, the nurse put them into category 2, but actually they belonged in category 1 as assigned by the physicians. For patients presenting with abdominal or flank pain, the physician placed them into category 3 if the chart did not record the severity of pain, but the nurses would place them into category 2 after seeing them in pain.
This study also shows that triage criteria have crucial limitations. Triage studies have stated important principles in the application of triage criteria, and 1 of them includes sensitive criteria which can find all potentially critical patients.(2)
Although the triage system did not perform well in predicting which patients required hospitalization, it is generally agreed that there are limitations in using hospital admissions as an outcome variable. Physicians' decisions to admit patients are often based on medical, social, and economic considerations. We understand that not all patients who require ED care require admission; however, at this time, admission is the only concrete outcome standard available.
The lack of physician-nurse agreement and the inability to predict outcomes have important implications for patient access to health care and for hospital planning. In general, because of differences in triage techniques and policies for hospital admission, it is essential to evaluate triage criteria in multiple emergency departments. Nevertheless, the findings in our study can serve as a future reference for improving triage criteria and the further analysis of triage systems.

Conclusions

We retrospectively evaluated a triage protocol in an emergency department. The physician-nurse inter-observer agreement with regards to triage decisions was poor. The lack of agreement on triage decision making has important implications for EDs in which the priority of care is based on nursing triage categorization.

Acknowledgments

This study was supported by grants from the Taiwan Emergency and Critical System Advanced Foundation. We would also like to thank the research assistant, Ms. Vicky F. Shih, and all members of the study team who assisted with data collection and processing.

REFERENCES

1. Beveridge R. The Canadian triage and acuity scale: a new and critical element in health care reform. J Emerg Med 1998;16:507-11.
2. Cain P, Waldrop RD, Jones J. Improved pediatric patient flow in a general emergency department by altering triage criteria. Acad Emerg Med 1996;3:65-71.
3. Read S, George S, Westlake L, Williams B, Glasgow J, Potter T. Piloting an evaluation of triage. Inter J Nurs Stud 1992;29:275-88.
4. Brillman JC, Doezema D, Tandberg D, Sklar DP, Skipper BJ. Does a physician's visual assessment change triage? Am J Emerg Med 1997;15:29-33.
5. Brillman JC, Doezema D, Tandberg D, Sklar DP, Davis KD, Simms S, Skipper BJ. Triage: limitations in predicting need for emergent care and hospital admission. Ann Emerg Med 1996;27:493-500.
6. George S, Read S, Westlake L, Fraser-Moodie A, Pritty P, Williams B. Differences in priorities assigned to patients by triage nurses and by consultant physicians in accident and emergency departments. J Epidemiol Commun Heal 1993;47:312-5.
7. Major Berman DA, Colonel Coleridge ST, McMurry TA. Computerized algorithm-directed triage in the emergency department. Ann of Emerg Med 1989;18:141-4.
8. Birnbaum A, Gallagher J, Utkewicz M, Gennis P, Carter W. Failure to validate a predictive model for refusal of care to emergency-department patients. Acad Emerg Med 1994;1:213-7.
9. Clemmer TP, Orme JF, Thomas F, Brooks KA. Prospective evaluation of the crams scale for triaging major trauma. J Trauma 1985;25:188-91.
10. Cook S, Sinclair D. Emergency Department Triage: a program assessment using the tools of continuous quality improvement. J Emerg Med 1997;15:889-94.
11. Derlet RW, Nishio D, Cole LM, Silva J. Triage of Patients Out of the Emergency Department: Three-Year Experience. Am J Emerg Med 1992;10:195-9.
12. Esposito TJ, Offner PJ, Jurkovich GJ, Griffith J, Maier RV. Do pre-hospital trauma center triage criteria identify major trauma victims? Arch Surg 1995;130:171-6.
13. Fries GR, McCalla G, Levitt MA, Cordova R. A prospective comparison of paramedic judgment and the trauma triage rule in the pre-hospital setting. Ann Emerg Med 1994;24:885-9.
14. Gill JM. Non-Urgent use of the emergency department: appropriate or not. Ann Emerg Med 1994;24:953-7.
15. Kane G, Engelhardt R, Celentano J, Koenig W, Yamanaka J, McKinney P, Brewer M, Fife D. Empirical development and evaluation of pre-hospital trauma triage instruments. J Trauma 1985;25:482-9.
16. Kilberg L, Clemmer TP, Clawson J, Wolley FR, Thomas F, Orme JF. Effectiveness of implementing a trauma triage system on outcome: a prospective evaluation. J Trauma 1988;28:1493-8.
17. Lowe RA, Bindman AB, Ulrich SK, Norman G, Scaletta TA, Keane D, Washington D, Grumbach K. Refusing care to emergency department patients: evaluation of published triage guidelines. Ann Emerg Med 1994;23:286-93.
18. Richardson D. No relationship between emergency department activity and triage categorization. Acad Emerg Med 1998;5:141-5.
19. Waldrop RD, Harper DE, Mandry C. Prospective assessment of triage in an urban emergency department. Southern Med J 1997;90:1208-12.
20. West JG, Murdock MA, Baldwin LC, Whalen E. A method for evaluating field triage criteria. J Trauma 1986; 26:655-9.
21. George SL, Read S, Westlake L, Williams B, Fraser-Moodie A, Pritty P. Evaluation of nurse triage in a British accident and emergency department. Brit Med J 1992; 304:876-8.
22. Foldes SS, Fischer LR, Kaminsky K. What is an emergency? The judgments of two physicians. Ann Emerg Med 1994;23:833-40.
23. Kennedy K, Aghababian RV, Gans L, Leuis CP. Triage: Techniques and applications in decision making. Ann Emerg Med 1996;28:136-44.

From the Department of Emergency Medicine, Chang Gung Memorial Hospital, Taipei; 1Department of Emergency Medicine, St. Paul's Hospital, Kweishan, Taoyuan.
Received: Jun. 1, 2001; Accepted: Mar. 29, 2002
Address for reprints: Dr. Song-Seng Loke, Department of Emergency Medicine, Chang Gung Memorial Hospital. 5, Fu-Shing Street, Kweishan, Taoyuan 333, R.O.C. Tel.: 886-3-3281200 ext. 2482; Fax: 886-3-3287715; E-mail: cgmhpcmd@ms4.hinet.net