INTRODUCTION

Self-reported disease history is often used in large-scale epidemiological studies, especially if no other data with higher validity are available. Although self-reported data can be useful, assessments of the reliability of self-reported data are inconsistent across studies, making researchers reluctant to utilize self-reported data sources. According to a study conducted in USA [1] assessing the validity of self-reported diagnoses in the first wave of the National Health and Nutritional Examination Survey Epidemiologic Follow-up Study, the positive predictive values (PPVs) for stroke and myocardial infarction (MI) were 67% and 83%, respectively. In contrast, lower validity was reported in a study conducted by Spanish researchers that investigated the Spanish cohorts of the European Prospective Investigation into Cancer and Nutrition study [2], which reported PPVs of 22.2% and 60.7% for stroke and MI, respectively. Moreover, a validation study for questionnaires answered by nurses [3] reported PPVs of 79% and 68% for cerebrovascular accidents and MI, which was not ideal considering the profession of the participants; this finding underscores the need for studies to validate each individual cohort. Several other studies from various countries showed PPVs for stroke and MI in the ranges of 29-79% and 43-81%, respectively.

In this study, we acquired hospital records from participants in a large-scale cohort study, the Health Examinees (HEXA) study, who self-reported stroke or MI and evaluated the validity of those participants’ self-reported disease history. In addition, we determined the level of agreement between specialists and non-specialists who reviewed the same medical records, in order to establish whether a specialist should review medical records for certain types of diseases.

METHODS

The HEXA study, one of the Korean Genome and Epidemiology Studies conducted by the Korean Centers for Disease Control and Prevention and known as KOGES_HEXA, is a population-based cohort study that aims to investigate the associations between epidemiological characteristics and various diseases, including stroke and MI [4]. A total of 173 357 participants were prospectively recruited between 2004 and 2013 at 38 health examination centers and training hospitals located in 8 different regions in Korea using a standardized study protocol [5]. Follow-up of the participants was started in 2007, comprising self-reports of being diagnosed with various diseases and changes in risk factors such as lifestyle habits. The validation of self-reported diseases through medical record review targeted participants who were followed during the period of 2012-2017. Details of the main objectives, rationale, study design, and baseline characteristics of the HEXA study have been published elsewhere [4].

To validate self-reported cases of disease, participants’ hospital records were reviewed and compared. For that purpose, written consent and delegation for acquiring hospital records, as well as a photocopy of an identification card, were collected from those who agreed to participate in this study. Six hired agents were assigned to make contact with participants and to acquire their hospital records, and were dispatched to different administrative regions throughout the country.

Among the followed participants in the HEXA cohort, 1488 subjects self-reported having been diagnosed with stroke or MI. We successfully made contact with 1338 of those subjects (89.9%) and asked whether each subject reconfirmed his or her prior self-report of disease and was willing to participate in this study. The numbers of subjects who reported no disease were 51 (11.8%) and 575 (54.5%) for stroke and MI, respectively. In total, 440 subjects reconfirmed their self-report, of whom 429 (28.8%) agreed to share their medical information. Those participants’ hospital records were acquired from the medical facility where the diagnosis was made.

Every acquired record was independently assigned to 2 medical doctors for review, and after that process, each record was labeled as ‘definite,’ ‘probable’, or ‘not’ stroke or MI. Only the records with specific test results indicating stroke or MI were labeled as ‘definite,’ such as patients with an abrupt onset of unilateral motor impairment with confirmatory magnetic resonance imaging results or patients with typically elevated ST segments on electrocardiography with elevated cardiac biomarkers. However, if the descriptions of the diagnostic tests did not clearly support the diagnosis, the reviewers looked for other definite evidence, such as hospital discharge records with a stroke or MI diagnosis and a referral for rehabilitation treatment. Medical records without definite evidence of the diagnosis of stroke or MI that nonetheless contained reasonable evidence that the patients might have been previously diagnosed with those conditions were labeled as ‘probable’; this category included patients who were transferred from another hospital after diagnosis and had several records of National Institutes of Health Stroke Scale scores or a history of rehabilitation. Those without any sign of either disease were classified as ‘not’ having stroke or MI. When the 2 doctors did not agree, a third doctor made the final judgement. Two PPVs were calculated: one only counted ‘definite’ disease cases as positive, and the other both included ‘probable’ and ‘definite’ disease cases as positive.

To derive proper guidelines for whether a specialist should review the medical records of participants in this cohort, the level of agreement between different doctors regarding the same medical records was observed. Thirty randomly selected cases of stroke and MI were given to a neurologist and a cardiologist, respectively, currently working at Seoul National University Hospital. Cohen’s kappa (κ) and the sensitivity and specificity were calculated, with the results from the neurologist and cardiologist used as the gold standard.

Ethics Statement

This study was approved by the Institutional Review Board of Seoul National University Hospital (C-0608-018-179).

RESULTS

The baseline characteristics of the participants in this study are shown in Table 1. Out of the 429 medical records that were reviewed, 57.2% and 69.7% were from male participants who self-reported stroke or MI, respectively. Most of the records (97.9% and 96.6%) were obtained from those who were 50 years old or older at the time of follow-up, but no participant was more than 80 years old. The participants were predominantly diagnosed in hospitals located in metropolitan areas such as Seoul, Busan, and Daegu.

Table 2 shows the results of the medical record review by 2 different medical doctors regarding self-reported stroke and MI, and the agreement between them is represented as the Cohen’s κ statistic. Of the 208 self-reported stroke cases and 221 MI cases, 154 (92 definite stroke, 4 probable stroke, and 58 not stroke) and 194 review results (57 definite MI, 1 probable MI, and 135 not MI) showed agreement between the 2 doctors. A total of 53 stroke and 28 MI records showed disagreement between the different doctors, and another medical doctor made the final judgement. The agreement between doctors, calculated as κ (95% confidence interval [CI]), was 0.54 (95% CI, 0.45 to 0.64) for stroke and 0.74 (95% CI, 0.66 to 0.82) for MI, which are moderate and substantial levels of agreement, respectively [6].

Figure 1 shows the results of the medical record review of self-reported stroke and MI. Out of the 208 self-reported stroke cases, 107 were determined to be definite stroke, 16 to be probable stroke, and 85 to be not stroke. Out of the 221 self-reported MI cases, 72 were classified as definite MI, 2 as probable MI, and 147 as not MI.

As shown in Table 3, the PPVs (95% CI) of self-reported stroke were 59.1% (95% CI, 52.3 to 65.6) and 51.4% (95% CI, 44.7 to 58.1) with and without the inclusion of probable strokes, respectively. The PPVs (95% CI) of self-reported MI were 33.5% (95% CI, 27.6 to 39.9) and 32.6% (95% CI, 26.7 to 39.0) with and without the inclusion of probable MI, respectively. The PPVs of males were slightly higher than those of females, but were not statistically significantly different. The PPVs for stroke and MI showed contrasting outcomes in different age groups, as the PPV for stroke decreased as age at follow-up increased, while the PPV for MI increased. When we stratified the results by the region of the hospital of diagnosis, no noticeable patterns emerged. Gyeongsang and Jeolla Provinces showed the highest PPVs for stroke, and Gangwon and Chungcheong Provinces showed the highest PPVs for MI.

Table 4 shows the agreement between different reviewers regarding 30 randomly selected medical records for each disease. For this process, we asked the doctors only to differentiate ‘disease’ from ‘not disease.’ Therefore, the ‘confirmed’ records would be equivalent to ‘definite’ only. The κ statistics (95% CI) for stroke and MI were 0.49 (95% CI, 0.18 to 0.81) and 0.46 (95% CI, 0.15 to 0.78), which both could be interpreted as moderate levels of agreement, while the sensitivity and specificity were both in the range of 71-78% [6].

DISCUSSION

We calculated PPVs to assess the validity of self-reported stroke and MI in questionnaires for the follow-up study of the HEXA cohort. The PPVs of our study participants were 51-59% for self-reported stroke and 33-34% for self-reported MI. For stroke, our result falls in the range reported in similar studies with large study populations previously performed in Europe and the USA [1,7-14], and was significantly higher than the PPVs reported in studies done in Spain and New Zealand [2,15]. However, the PPV for MI in our study was far lower than those in other similar studies [1-3,7,8,10,12,13,16-19]. The only study that showed a similar PPV for MI (43%) was a study conducted in Japan among the Japan Public Health Center-based (JPHC) prospective study cohort.

An important fact worth mentioning that could explain the low PPV for MI is that the follow-up questionnaire participants in HEXA between 2012 and 2016 were asked about either MI or angina. Because MI was our only outcome of interest in this study, we asked the participants whether they had been diagnosed with MI and only included those who reconfirmed the diagnosis. This resulted in a large proportion of subjects who reported no MI, despite previously self-reporting MI on follow-up. However, we included participants who were not sure whether they were diagnosed with MI or angina when re-contact was made as candidates for medical record review. This possibility may be supported by the fact that a large proportion of false-positive cases turned out to be angina. Moreover, a few of the acquired hospital records were not related to the diseases of interest. Those records were therefore classified as not confirming the disease, resulting in a lower PPV; this may have taken place because study participants cannot always precisely recall the date and place of their first diagnosis.

Nonetheless, the higher PPV observed for stroke than for MI could have reflected the distinct characteristics of these diseases, as stroke is typically accompanied by very noticeable and memorable symptoms such as motor impairment or prolonged dysphasia, while MI and other cardiovascular diseases share symptoms such as pain or discomfort in the chest. Moreover, the follow-up questionnaire for stroke only asked about stroke, and no irrelevant cases were included in the candidates for medical record review.

The finding of higher PPVs in males than in females in our study is consistent with the findings of other studies from the USA and Japan [14,19], despite its statistical insignificance. The observation that the PPV for stroke was inversely related with age at follow-up is also in accordance with the findings reported for the JPHC cohort; this trend could be the result of cognitive decline or cortical dysfunction in elderly individuals [14]. However, a previous study performed in the Netherlands among only individuals aged 85 and over showed one of the highest PPVs for stroke (78%) of the various studies that have investigated this issue, even though the total number of subjects was small (n=9) [17].

In other words, only half of the patients correctly gave their stroke diagnosis on questionnaires during follow-up and in regard to MI, only one-third of the responses were true in our study, regardless of whether the results included possible disease. Such a high rate of false positives for patients’ outcomes in a cohort not only distorts the descriptive statistics in this population, but also poses the risk of greatly misleading subsequent studies that utilize this dataset. Therefore, self-reported disease history in HEXA should be supported by linkage to other reliable data, such as National Health Insurance claims data, and should not be used alone.

In terms of the agreement between different reviewers and between general doctors and specialists, the calculated κ values were 0.49 and 0.46 for cases of stroke and MI, both showing moderate levels of agreement. The characteristics of the 30 cases each used for the comparisons between specialists and general doctors were not statistically significant different from those of the overall subjects. Although it is undoubtedly best for a specialist, such as a neurologist, to review hospital records for relevant conditions, doing so is not always possible, nor is it always the most efficient approach. According to our results, only 2 and 4 cases of stroke and MI, respectively, were classified by specialists as ‘not disease’ despite being classified as ‘disease’ by non-specialist doctors, whereas it was more common for specialists to reclassify cases that were initially categorized as ‘not disease’ by non-specialist doctors. This discrepancy implies that only the negative results of medical record reviews performed by non-specialists need revision by a specialist. Furthermore, to improve the accuracy and consistency of the identification of disease, a standardized protocol for acquiring and reviewing medical records is recommended, including which subcategory to review in a record and which diagnostic criteria to use.

The importance of this study lies in the fact that it is the first study to investigate the validity of self-reported stroke and MI in the Korean population, as the validity of self-reporting differs widely according cultural characteristics, such as language. Moreover, we provided meaningful results regarding the agreement between doctors of different specialties who reviewed the same medical records and the possibility of implementing more efficient and better processes when reviewing medical records in future studies.

A few limitations of this study should be noted. First, not all the hospital records were acquired from patients who self-reported stroke or MI during follow-up, making it impossible to measure the sensitivity or specificity of self-reporting, such that the results were restricted to PPVs. Second, information on important factors that may affect the validity of self-reported diseases, such as socioeconomic status or education level, was not available, leaving unresolved questions about which variables may be correlated with validity. In addition, only 429 cases (28.8%) were reviewed for validity out of 1488 self-reported cases. It was frequently the case either that it was not possible to contact the subject or that the subject was not willing to participate and refused to provide consent for the acquisition of his or her medical records. The limited number of cases reviewed could have resulted in selection bias because the characteristics of the cases that were reviewed or not reviewed could have been different. Nevertheless, we could not address these possible differences because we had limited information on the study participants who refused to provide consent for medical record review.

In conclusion, the PPVs of self-reported stroke and MI were 59.1% and 33.5%, respectively, which were lower than expected, and inter-reviewer agreement was moderate to substantial. Proper review of hospital data is needed to validate self-reported disease history prior to utilizing it in further studies.

Notes

CONFLICT OF INTEREST

The authors have no conflicts interests associated with the material presented in this paper.

ACKNOWLEDGEMENTS

Notes

AUTHOR CONTRIBUTIONS

Conceptualization: SC, AS, JKL, DK. Data curation: SC, JL, JL. Formal analysis: SC, JL. Funding acquisition: JKL. Methodology: AS, JKL, DK. Project administration: JL. Visualization: SC. Writing - original draft: SC, AS. Writing - review & editing: SC, JL, JL, DK, JKL, AS.

Figure. 1.

Results of medical record review of self-reported stroke and myocardial infarction (MI).

REFERENCES

• 1. Bergmann MM, Byers T, Freedman DS, Mokdad A. Validity of self-reported diagnoses leading to hospitalization: a comparison of self-reports with hospital records in a prospective study of American adults. Am J Epidemiol 1998;147(10):969-977ArticlePubMedPDF
• 2. Machón M, Arriola L, Larrañaga N, Amiano P, Moreno-Iribas C, Agudo A, et al. Validity of self-reported prevalent cases of stroke and acute myocardial infarction in the Spanish cohort of the EPIC study. J Epidemiol Community Health 2013;67(1):71-75ArticlePubMed
• 3. Colditz GA, Martin P, Stampfer MJ, Willett WC, Sampson L, Rosner B, et al. Validation of questionnaire information on risk factors and disease outcomes in a prospective cohort study of women. Am J Epidemiol 1986;123(5):894-900ArticlePubMedPDF
• 4. Kim Y, Han BG; KoGES group. Cohort profile: the Korean Genome and Epidemiology Study (KoGES) consortium. Int J Epidemiol 2017;46(4):1350ArticlePubMedPMCPDF
• 5. Cho S, Shin A, Song D, Park JK, Kim Y, Choi JY, et al. Validity of self-reported cancer history in the health examinees (HEXA) study: a comparison of self-report and cancer registry records. Cancer Epidemiol 2017;50(Pt A):16-21ArticlePubMed
• 6. Landis JR, Koch GG. The measurement of observer agreement for categorical data. Biometrics 1977;33(1):159-174ArticlePubMed
• 7. Barr EL, Tonkin AM, Welborn TA, Shaw JE. Validity of self-reported cardiovascular disease events in comparison to medical record adjudication and a statewide hospital morbidity database: the AusDiab study. Intern Med J 2009;39(1):49-53ArticlePubMed
• 8. Eliassen BM, Melhus M, Tell GS, Borch KB, Braaten T, Broderstad AR, et al. Validity of self-reported myocardial infarction and stroke in regions with Sami and Norwegian populations: the SAMINOR 1 Survey and the CVDNOR project. BMJ Open 2016;6(11):e012717ArticlePubMedPMC
• 9. Engstad T, Bonaa KH, Viitanen M. Validity of self-reported stroke: the Tromso Study. Stroke 2000;31(7):1602-1607ArticlePubMed
• 10. Haapanen N, Miilunpalo S, Pasanen M, Oja P, Vuori I. Agreement between questionnaire data and medical records of chronic diseases in middle-aged and elderly Finnish men and women. Am J Epidemiol 1997;145(8):762-769ArticlePubMedPDF
• 11. Jamrozik E, Hyde Z, Alfonso H, Flicker L, Almeida O, Yeap B, et al. Validity of self-reported versus hospital-coded diagnosis of stroke: a cross-sectional and longitudinal study. Cerebrovasc Dis 2014;37(4):256-262ArticlePubMed
• 12. Misirli G, Bamia C, Dilis V, Benetou V, Zilis D, et al. Validation of self-reported incident cardiovascular disease events in the Greek EPIC cohort study. Ital J Public Health 2012;9(3):e7538
• 13. Tretli S, Lund-Larsen PG, Foss OP. Reliability of questionnaire information on cardiovascular disease and diabetes: cardiovascular disease study in Finnmark county. J Epidemiol Community Health 1982;36(4):269-273ArticlePubMedPMC
• 14. Yamagishi K, Ikeda A, Iso H, Inoue M, Tsugane S; JPHC Study Group. Self-reported stroke and myocardial infarction had adequate sensitivity in a population-based prospective study JPHC (Japan Public Health Center)-based Prospective Study. J Clin Epidemiol 2009;62(6):667-673ArticlePubMed
• 15. Carter K, Barber PA, Shaw C. How does self-reported history of stroke compare to hospitalization data in a population-based survey in New Zealand? Stroke 2010;41(11):2678-2680ArticlePubMed
• 16. Bush TL, Miller SR, Golden AL, Hale WE. Self-report and medical record report agreement of selected medical conditions in the elderly. Am J Public Health 1989;79(11):1554-1556ArticlePubMedPMC
• 17. Lagaay AM, van der Meij JC, Hijmans W. Validation of medical history taking as part of a population based survey in subjects aged 85 and over. BMJ 1992;304(6834):1091-1092ArticlePubMedPMC
• 18. Paganini-Hill A, Chao A. Accuracy of recall of hip fracture, heart attack, and cancer: a comparison of postal survey data and medical records. Am J Epidemiol 1993;138(2):101-106ArticlePubMedPDF
• 19. Rosamond WD, Sprafka JM, McGovern PG, Nelson M, Luepker RV. Validation of self-reported history of acute myocardial infarction: experience of the Minnesota Heart Survey Registry. Epidemiology 1995;6(1):67-69ArticlePubMed

Figure & Data

References

Citations

Citations to this article as recorded by  

• Validation of self-reported cardiovascular problems in childhood cancer survivors by contacting general practitioners: feasibility and results
  Eva-Maria Hau, Tomáš Sláma, Stefan Essig, Gisela Michel, Laura Wengenroth, Eva Bergstraesser, Nicolas X. von der Weid, Christina Schindera, Claudia E. Kuehni
  BMC Primary Care.2024;[Epub]     CrossRef
• Stressful life events and heart disease and stroke: A study among Portuguese older adults
  Ana Quaresma, Elisabete Alves, Silvia Fraga, Ana Henriques
  Stress and Health.2023;[Epub]     CrossRef
• Association of the amount of alcohol consumption with change in skeletal muscle and fat mass among Korean adults
  Jihun Song, Seulggie Choi, Sun Jae Park, Sung Min Kim, Yoosun Cho, Gyeongsil Lee, Ji Soo Kim, Sang Min Park
  Alcohol.2023; 111: 9.     CrossRef
• History of Breastfeeding and Stroke Prevalence Among Parous Females: Findings from the National Health and Nutrition Examination Survey, 1999–2012
  Gina M. Richardson, Emily Mitchell, Kalera Stratton, Laura R. Kair, Lynn M. Marshall
  Breastfeeding Medicine.2022; 17(9): 768.     CrossRef
• Joint effect of elevated-c-reactive protein level and hypertension on new-onset stroke: A nationwide prospective cohort study of CHARLS
  Xuanli Chen, Siyuan Liu, Jiadong Chu, Wei Hu, Na Sun, Yueping Shen
  Frontiers in Public Health.2022;[Epub]     CrossRef
• Objectively Measured Built Environments and Cardiovascular Diseases in Middle-Aged and Older Korean Adults
  Eun Young Lee, Jungsoon Choi, Sugie Lee, Bo Youl Choi
  International Journal of Environmental Research and Public Health.2021; 18(4): 1861.     CrossRef