Lee, Kim, Lee, and Shin: Reliability and Validity of the Appropriateness Evaluation Protocol for Public Hospitals in Korea

### Objectives

This study was conducted to assess the applicability of the Appropriateness Evaluation Protocol (AEP) for public hospitals in Korea.

### Methods

In May 2016, 1500 admission claims were collected from Korean public district hospitals using stratified random sampling. Of these claims, 560 admissions to 37 hospitals were retrieved for analysis. Medical records administrators determined the appropriateness of admission using the criteria detailed in the AEP, and a physician separately assessed the appropriateness of admission based on her clinical judgment. To examine the applicability of the AEP, the concordance of the decisions made between a pair of AEP reviewers and between an AEP reviewer and a physician reviewer was compared.

### Results

The results showed an almost perfect inter-rater agreement between the AEP reviewers and a moderate agreement between the AEP reviewers and the physician. The sensitivity and specificity of the AEP were calculated as 0.86 and 0.56, respectively.

### Conclusions

Our findings suggest that the AEP could potentially be applied to Korean public hospitals as a reliable and valid instrument for assessing the appropriateness of admissions.

### INTRODUCTION

The appropriateness of hospital admission is a major healthcare issue around the world, as it is closely related to the quality of healthcare services and healthcare costs. However, the standards for determining the appropriateness of hospitalization vary depending on the perspectives of different stakeholders, and objective criteria to assess hospitalization appropriateness are difficult to develop [1,2]. To address this problem, many researchers have created evaluation tools such as the InterQual criteria, the Appropriateness Evaluation Protocol (AEP), and the Managed Care Appropriateness Protocol [3-5]. Of these, the AEP, which was developed by Gertman and Restuccia [4] in 1981, is the most widely used. This protocol is known for its ease of use and efficiency compared to similar assessments; a study found that the average time taken to assess an admission using the AEP was less than 10 minutes. The original AEP consists of 18 criteria across 2 sections. The medical necessity of admission is determined by assessing the condition of the patient and the clinical services, if any, that the patient requires. An admission must fulfill at least 1 of the 18 criteria to be identified as medically necessary [4]. The AEP has been validated in the USA and elsewhere [4,6-9], and it has been applied in many countries because of the high-level of objectivity and adaptability of its criteria [6,10-16].
In Mexico, the AEP criteria have been applied to evaluate the reliability and validity of the AEP regarding the appropriateness of admissions and hospital stays in elderly patients. In that study, inter-reviewer agreement had a kappa coefficient of >0.70. Although sensitivity and positive predictive value (used to detect inappropriate admissions) were not calculated, specificity and negative predictive value (used to detect appropriate admissions) were found to be >94.0% and >98.0%, respectively [17].
In the UK, the American version of the Paediatric AEP was reviewed by a panel of pediatricians and general practitioners. In that study, the agreement between the raters using the Paediatric AEP for admission criteria was excellent (κ=0.848), whereas there was poor agreement (κ=0.345) between clinicians using subjective judgment [18].
Validation of the AEP in previous studies was mostly conducted in teaching hospitals using a small sample (e.g., 1 or 2 teaching hospitals). In Asia, only a few countries, such as China, Malaysia, and Hong Kong, have applied the AEP to evaluate the appropriateness of hospital admissions [7,19,20]. Several studies have assessed the use of the AEP criteria to evaluate the appropriateness of hospitalization in Korea. However, those studies focused on specific clinical departments or patients with certain diseases [21,22].
The public sector accounts for fewer than 10% of Korea’s medical institutions. Recently, calls have been made to increase the proportion of public medical institutions due to the public’s demand for the national health insurance system to provide expanded coverage for their basic health needs and to the need for government intervention in the case of market failure. In this respect, it is necessary to review the quality and performance of Korean public healthcare services. The objective of this study was to assess the appropriateness of hospital admissions using the AEP and to validate the applicability of the AEP for public hospitals.

### Source of the Data Under Study

We utilized hospital admission records collected by the Health Insurance Review and Assessment Service (HIRA), an affiliate of the Ministry of Health and Welfare in Korea, in our attempt to validate the AEP. To monitor the performance of public district hospitals, HIRA collected the records of all patients admitted in May 2016 to a public district hospital (n=39) and stored them as scanned files. Public district hospitals are located thr-oughout the country to provide medical and surgical care to local communities. They also offer public health services that are not provided by private hospitals [23].

### Validation of the Appropriateness Evaluation Protocol

We attempted to validate the AEP by comparing the appropriateness of hospital admission determined by the AEP with the appropriateness determined by an experienced physician. The physician’s decision, in other words, served as the gold standard.
We trained 2 professional medical record administrators to apply the original US-AEP to review the 560 admission records selected from the HIRA dataset and determine whether admission was appropriate. If there was a discrepancy between the 2 medical record administrators, an experienced nurse would use the same AEP criteria to make the final determination. The physician whose judgment served as the gold standard was a faculty member in the Department of Family Medicine at a major tertiary teaching hospital and had more than 10 years of clinical experience. She reviewed all 560 admission records and determined whether each admission was appropriate based on her clinical judgment. All of the reviews were carried out concurrently and independently.
As a test of the reliability of the AEP, we examined the degree of agreement between the decisions on admission appropriateness made by the 2 medical record administrators. The degree of agreement was measured by calculating the overall agreement and the specific agreement. The overall agreement referred to the proportion of decisions in which the 2 reviewers agreed. The specific agreement was separated into 2 components: specific appropriate agreement (referring to the proportion of agreement in cases where admission is judged to be appropriate by at least 1 reviewer) and specific inappropriate agreement (referring to the proportion of agreement in cases where admission is determined to be inappropriate by at least 1 reviewer). To take into account the probability of chance agreement, Cohen’s kappa statistic was calculated for overall agreement. The kappa statistic was interpreted using the guidelines set forth by Landis and Koch [24], in which kappa coefficients between 0.00-0.20 are regarded as slight agreement, between 0.21-0.40 as fair agreement, between 0.41-0.60 as moderate agreement, between 0.61-0.80 as substantial agreement, and between 0.81-1.00 as almost perfect agreement.
The validity of the AEP was tested by comparing the final decisions made by the AEP reviewers with the physician’s judgment regarding the appropriateness of admission. The overall agreement and specific agreement were calculated to measure the level of concordance between the AEP and the gold standard. The level of overall agreement was presented using Cohen’s kappa values. In addition, we calculated sensitivity, specificity, and positive and negative predictive values.
All statistical analyses were performed using SAS version 9.3 (SAS institute Inc., Cary, NC, USA).

### Ethics Statement

The present study protocol was reviewed and approved by the Institutional Review Board of the College of Medicine, The Catholic University of Korea (approval No. MC19EESI0014). Informed consent was obtained from all subjects at the time of their enrollment in the study.

### RESULTS

Of the 560 admission records reviewed in the study, 424 (75.7%) represented patients who were admitted to medical wards, 128 (22.9%) to surgical wards, and 8 (1.4%) to other wards. In terms of hospital type, 503 patients (89.8%) were admitted to general hospitals (i.e., hospitals that have over 100 beds and more than 7 departments – surgery, medicine, pediatrics, radiology, anesthesiology, etc.), and the remaining 57 patients (10.2%) were admitted to small hospitals. The distribution of patients in the study sample by department (medical, surgical, or other) is shown in Table 1.
Of the 560 admissions, 138 were assessed to be inappropriate using the AEP criteria (24.6%). The proportion of inappropriate admissions was 20.5% and 39.8% for medical wards and surgical wards, respectively (Table 1).
The level of overall agreement between the 2 AEP reviewers was 96.8%, and the agreement values obtained were 97.9% and 93.4% for appropriate and inappropriate admissions, respectively. The Cohen’s kappa value for overall agreement was 0.91 (95% confidence interval [CI], 0.87 to 0.95). According to the Landis and Koch [24] guidelines, the kappa statistics show-ed almost perfect agreement between the 2 AEP reviewers (Table 2).
The level of overall agreement between the AEP reviewers and the physician was 78.6%. Of the 560 total admissions, 360 admissions were assessed to be appropriate and 80 admissions were assessed to be inappropriate by both the AEP and the physician reviewers. The agreement values for appropriate and inappropriate admissions were 85.7% and 57.1%, respectively. Of the 418 admissions that were judged to be appropriate by a physician reviewer, 58 admissions (13.9%) were judged to be inappropriate using the AEP. Of the 142 physician-judged inappropriate admissions, 62 admissions (43.7%) were judged to be appropriate using the AEP. The Cohen’s kappa coefficient for the level of overall agreement between the AEP reviewers and the physician was 0.43 (95% CI, 0.34 to 0.51). According to the Landis and Koch [24] guidelines, a kappa coefficient of 0.42 indicates a moderate agreement on the appropriateness of admissions between reviewers using AEP criteria and the judgment of an expert. The sensitivity and specificity of the AEP were found to be 0.86 and 0.56, respectively. The positive and negative predictive values were 0.85 and 0.58, respectively (Table 3).

### DISCUSSION

In this study, 24.6% of the admissions to public district hospitals in Korea under consideration were identified to be inappropriate based on the AEP criteria. Previous studies from Italy, Portugal, Iran, China, Denmark, and France reported that the rate of inappropriate admissions ranged from 8.8% to 28.1% [7,10,13,25-27]. In comparison, the rate of inappropriate hospital admissions in Korea is high, suggesting the importance of finding a valid and reliable tool to assess admission appropriateness. With such a tool, researchers can investigate the causes of inappropriate hospital admissions to help policymakers and healthcare leaders combat the problem.
This study showed the AEP to have high reliability and high validity. In particular, the level of overall agreement between the 2 AEP reviewers was found to be almost perfect (96.8%), with a kappa value of 0.91. These findings were similar to the values reported in studies conducted in Europe, China, Iran, Italy, and Hong Kong [6-8,10,20]. Regarding validity, a moderate level of agreement was found between the AEP reviewers and the physician whose clinical judgment served as a gold standard. The degree of overall agreement (78.6% with a kappa value of 0.43) obtained was lower than that found in studies conducted in Iran and China [6,7]. Esmaili et al. [6] tested the reliability and validity of the AEP for determining the appropriateness of hospital admissions in one of the largest Iranian teaching hospitals. The level of overall agreement between the AEP reviewers and the physician reviewers was 92.0%, with a kappa value of 0.80. Liu et al. [7] validated the AEP in China using hospital records from 2 tertiary hospitals. Of 350 randomly selected admissions, the level of overall agreement between the AEP reviewers and the physician reviewers was 90.9%, with a kappa value of 0.68.
Restricted access to hospital records allowed us to obtain admission records from public hospitals for only 1 month of the year. Given the fact that the study period was limited to May 2016, we cannot guarantee that no seasonal variation exists in hospital admission data. However, we believe that this is not a major problem, considering that our study selected the most medical records available, thereby ensuring that the data included patients with an adequately wide range of conditions.
In addition to the limitations associated with the 1 month admission data window, another limitation of this study was that there was only 1 physician reviewer. Clinicians often decide whether patients should be hospitalized based on the present illness, the patient’s overall condition, and patient information and objective figures. Korea’s hospital system is less burdened by patients than other countries, and cultural differences may make it difficult to define the AEP as an admission standard.
In this study, some admission records were excluded after random sampling. Pediatric admission data were excluded from the study because pediatric cases have different hospital admission criteria than adult cases. Psychiatric admission data were also excluded because in those cases, the patient’s condition or the severity of the disease were not documented in detail. Finally, cases with different hospitalization periods or with invalid or incomplete charts were removed from the data set. In total, 560 cases were reviewed. This exclusion of certain data from the study may have compromised randomization.
Since the analyses were done solely on data obtained from public district hospitals, the absence of private hospital admission data means that these results may not be fully representative of Korean hospitals in general. Thus, further research is needed that includes data from hospitals from both the public and private sectors. However, there are many considerations associated with the establishment of a system for collecting medical records, such as the confidentiality of personal information and privacy issues.
Another limitation of this study is the absence of data on the healthcare costs and the physician workload associated with inappropriate admissions. It is widely believed that inappropriate hospitalization may have a negative impact on medical costs and the healthcare workforce. Several studies have been published on the economic cost of inappropriate admissions [26,29,30]. Menand et al. [27] conducted a multicenter study including 8 hospitals in France and showed that the median cost of a hospital stay was higher for inappropriate admissions (€4399.2) than for appropriate admissions (€3606.5). Mould-Quevedo et al. [29] also found that the cost of inappropriate hospitalization per patient (US$2323.3) was about 1.6 times higher than the cost of appropriate hospitalization (US$1497.2) among elderly patients (over 60 years old) admitted to a general hospital in Mexico City. Nevertheless, due to constraints on data availability, the cost of inappropriate admissions and associated factors were not explored in this study. Future studies should aim to collect information on these factors associated with the financial implications of the AEP and to perform in-depth analysis of these data.
The main limitations of the current study were limited access to medical records from private hospitals and an absence of data on the healthcare costs and physician workload associated with inappropriate admissions. As topics of future research, determination of the potential implications of adopting the AEP in terms of admissions per bed and the workload of physicians and nursing staff would provide insights into the efficiency aspect of the AEP as an appropriateness evaluation tool. Additionally, examining the effectiveness of educating reviewers with non-clinical backgrounds on the clinical knowledge needed to use the AEP to is suggested for future research.
In conclusion, our findings show that the AEP is a reliable instrument for determining the appropriateness of admissions in public district hospitals in Korea. However, due to differences in clinical practice and hospital admission policies between Korea and the USA, the AEP should be modified for use in Korea.

### CONFLICT OF INTEREST

The authors have no conflicts of interest associated with the material presented in this paper.

### Notes

AUTHOR CONTRIBUTIONS
Conceptualization: ES. Data curation: SJHK. Formal analysis: SJHK, CL. Funding acquisition: None. Methodology: CL, ES. Project administration: CL, SJHK. Visualization: SJHK. Writing - original draft: CL, SJHK. Writing - review & editing: CL, ES.

### ACKNOWLEDGMENTS

None.
##### Table 1.
Distribution of study subjects across departments and proportion of appropriate and inappropriate admissions by department
Appropriate Inappropriate
Medical 424 (75.7) 337 (79.5) 87 (20.5)
Surgical 128 (22.9) 77 (60.2) 51 (39.8)
Other 8 (1.4) 8 (100) 0 (0.0)
Total 560 (100) 422 (75.4) 138 (24.6)

Values are presented as number (%).

##### Table 2.
Inter-rater agreement between two AEP reviewers
Reliability measure All departments, %
Agreement
Overall 96.8
Cohen’s kappa (95% CI) for overall agreement 0.91 (0.87, 0.95)*

AEP, Appropriateness Evaluation Protocol; CI, confidence interval.

* p<0.05.

##### Table 3.
Validity measures of the AEP on the appropriateness of hospital admissions when compared with the judgment of a physician (used as the gold standard)
Validity measure Validity across all departments, %
Agreement
Overall 78.6
Appropriated 85.7
Inappropriate 57.1
Cohen’s kappa (95% CI) for overall agreement 0.43 (0.34, 0.51)*
Sensitivity 86.1
Specificity 56.3
Predictive value
Positive 85.3
Negative 58.0

AEP, Appropriateness Evaluation Protocol; CI, confidence interval.

* p<0.05.

### REFERENCES

1. Donabedian A. The quality of care. How can it be assessed? JAMA 1988;260(12):1743-1748.

2. Lohr KN, Schroeder SA. A strategy for quality assurance in Medicare. N Engl J Med 1990;322(10):707-712.

3. Change Healthcare. InterQual® level of care criteria. [cited 2019 Mar 5]. Available from: https://www.changehealthcare.com/solutions/interqual/level-of-care-criteria.

4. Gertman PM, Restuccia JD. The appropriateness evaluation protocol: a technique for assessing unnecessary days of hospital care. Med Care 1981;19(8):855-871.

5. Oak Group. Criteria. [cited 2019 Mar 7]. Available from: https://oakgroup.com/mcap-clinical-utilisation-review/clinical-utilisation-review-criteria/.

6. Esmaili A, Ravaghi H, Seyedin H, Delgoshaei B, Salehi M. Developing of the appropriateness evaluation protocol for public hospitals in Iran. Iran Red Crescent Med J 2015;17(3):e19030.

7. Liu W, Yuan S, Wei F, Yang J, Zhang Z, Zhu C, et al. Reliability and validity of the Chinese version appropriateness evaluation protocol. PLoS One 2015;10(8):e0136498.

8. Lorenzo S, Lang T, Pastor R, Tampieri A, Santos-Eggimann B, Smith H, et al. Reliability study of the European appropriateness evaluation protocol. Int J Qual Health Care 1999;11(5):419-424.

9. Robain M, Lang T, Fontaine A, Logerot H, Monnet E, Six P, et al. Reliability and validity of the French version of the first part of the appropriateness evaluation protocol (AEPf): pertinent criteria of hospitalization stay. Rev Epidemiol Sante Publique 1999;47(2):139-149.

10. Apolone G, Fellin G, Tampieri A, Bonanoni E, Crosti PF, Lanzi E, et al. Appropriateness of hospital use: report from an Italian study. Eur J Public Health 1997;7(1):34-39.

11. Kaya S, Vural G, Eroğlu K, Sain G, Mersin H, Karabeyoğlu M, et al. Liability and validity of the appropriateness evaluation protocol in Turkey. Int J Qual Health Care 2000;12(4):325-329.

12. Lang T, Liberati A, Tampieri A, Fellin G, Gonsalves Mda L, Lorenzo S, et al. A European version of the appropriateness evaluation protocol: goals and presentation. Int J Technol Assess Health Care 1999;15(1):185-197.

13. Jepsen HK, Hendriksen C, Nielsen H, Nybo B, Perrild H. Every seventh acute medical admission is preventable. Dan Med J 2013;60(3):A4595.

14. Panis LJ, Verheggen FW, Pop P. To stay or not to stay. The assessment of appropriate hospital stay: a Dutch report. Int J Qual Health Care 2002;14(1):55-67.

15. Sangha O, Schneeweiss S, Wildner M, Cook EF, Brennan TA, Witte J, et al. Metric properties of the appropriateness evaluation protocol and predictors of inappropriate hospital use in Germany: an approach using longitudinal patient data. Int J Qual Health Care 2002;14(6):483-492.

16. Santos-Eggimann B, Paccaud F, Blanc T. Medical appropriateness of hospital utilization: an overview of the Swiss experience. Int J Qual Health Care 1995;7(3):227-232.

17. Sánchez-García S, Juárez-Cedillo T, Mould-Quevedo JF, García-González JJ, Contreras-Hernández I, Espinel-Bermudez MC, et al. The hospital appropriateness evaluation protocol in elderly patients: a technique to evaluate admission and hospital stay. Scand J Caring Sci 2008;22(2):306-313.

18. Esmail A. Development of the paediatric appropriateness evaluation protocol for use in the United Kingdom. J Public Health Med 2000;22(2):224-230.

19. Chan SC. Appropriateness of medical admissions from a Malaysian public primary care clinic. Med J Malaysia 1999;54(3):329-337.

20. Leung LP, Fan KL. Who should be admitted to hospital? Evaluation of a screening tool. Hong Kong Med J 2008;14(4):273-277.

21. Hwang JI. Characteristics of patient and healthcare service utilization associated with inappropriate hospitalization days. J Adv Nurs 2007;60(6):654-662.

22. Yoem HY, Kim SL. Appropriateness evaluation of hospitalization for the cerebral ischemia patients. J Korean Acad Community Health Nurs 1999;10(1):80-92 (Korean).

23. Oh J. Modernization of public district general hospitals: 2011 modularization of Korea’s development experience. Seoul: Ministry of Strategy and Finance; 2012. p. 19-21.

24. Landis JR, Koch GG. The measurement of observer agreement for categorical data. Biometrics 1977;33(1):159-174.

25. Tavakoli N, Hosseini Kasnavieh SM, Yasinzadeh M, Amini M, Mahmoudi Nejad M. Evaluation of appropriate and inappropriate admission and hospitalization days according to appropriateness evaluation protocol (AEP). Arch Iran Med 2015;18(7):430-434.

26. Gamper G, Wiedermann W, Barisonzo R, Stockner I, Wiedermann CJ. Inappropriate hospital admission: interaction between patient age and co-morbidity. Intern Emerg Med 2011;6(4):361-367.

27. Menand E, Lenain E, Lazarovici C, Chatellier G, Saint-Jean O, Somme D, et al. French multicenter evaluation of the appropriateness of admission to the emergency department of the over-80s. J Nutr Health Aging 2015;19(6):681-687.

28. Andrew MK, Powell C. An approach to ‘the social admission’. Can J Gen Intern Med 2015;10(4):20-22.

29. Mould-Quevedo JF, García-Peña C, Contreras-Hernández I, Juárez-Cedillo T, Espinel-Bermúdez C, Morales-Cisneros G, et al. Direct costs associated with the appropriateness of hospital stay in elderly population. BMC Health Serv Res 2009;9: 151.

30. Rojas-García A, Turner S, Pizzo E, Hudson E, Thomas J, Raine R. Impact and experiences of delayed discharge: a mixed-studies systematic review. Health Expect 2018;21(1):41-56.

TOOLS
Summary
Full text via DOI
CrossRef TDM
E-Mail
Print
Share:
METRICS
 0 Crossref
 0 Scopus
 2,746 View