INTRODUCTION

The concept of e-health development (Unified Health Information System) was implemented in Kazakhstan in 2013-2020, aimed at introducing an electronic health passport and a medical database. These tasks were designed to provide a logistics system for storing and exchanging key human health data, contribute to the implementation of urgent healthcare tasks, and improve management at all levels. The aim of these processes was to create a single information space for all interested parties (potential users of information): various structures and health services, management and control bodies, manufacturers of medical equipment and medicines, research organizations, and consumers of medical goods and services. This would significantly intensify the exchange of information and hasten the speed of introducing the latest scientific advances into everyday practice to meet the challenges of improving and developing healthcare. New types of information technology (IT) allow significant improvements in the efficiency of management and solve complex health problems through prompt access to specialized databases [1].

The choice of methods for assessing the effectiveness of IT is complicated by the fact that it is not always clear what kind of efficiency needs to be emphasized. For example, the classical approach to performance assessment considers overall efficiency from the standpoint of its 3 main components: organizational, social, and economic efficiency. Within the framework of organizational efficiency, many experts suggest assessing the technological effect of the introduction of IT, and the criteria for assessing this type of effect include manufacturability, novelty, reliability, simplicity, flexibility, increased productivity, and so forth. Social efficiency is implemented in the form of fulfilling the expectations, needs, and interests of employees, customers, and partners of the organization (good salary, pleasant working conditions, opportunities for personal development; no queues during service, etc.). Economic (financial, budgetary) efficiency is determined by the ratio of costs and results in value terms [2]. In non-medical fields, several balance methods are quite actively used, such as cost-benefit analysis, total cost of application ownership, total benefits of ownership, IT integration risk, information economics, critical success factors, balanced scorecard, and total economic impact.

This category also includes factor analysis methods, which are usually used to manage the entire organization, determine its business strategy, and assess the effectiveness of management based on information, such as real option valuation, applied information economics, and information technologies portfolio management. The balanced score strategic management assessment methodology can be singled out, which can also be used to assess the effectiveness of IT applications [3,4]. Different IT applications of the same type can bring about distinct effects and have different significance, which has elicited interest in the implementation of IT. The size of the effect varies among different groups of users and can be assessed from the perspectives of product (for example, improving the quality and increasing the range of information services produced), technology (increasing labor productivity), functionality (increasing management efficiency and optimizing organizational structure), and social effects (improving the quality of services and working conditions). Up to 50 IT efficiency monitoring parameters are given in the literature, with examples including productivity, customer satisfaction, embeddedness in business; analysis of additional and service works and costs, use of residual resources, utilization, regeneration, decomposition [5].

Individual experts recommend the following criteria for IT effectiveness: efficiency (the degree of achievement of tasks) and quality (the degree of compliance with requirements, specifications, and expectations). For example, the attributes of software quality include functionality, an intuitive interface, fault tolerance, scalability, ability to change configuration, portability, and reliability [6]. A well-known study, Kano analysis, has highlighted the main requirements that users have for system-wide and office software, based on interviews with organizations. The following characteristics were assessed: familiarity with the software, user-friendliness, ease of use, speed of operation, stability of operation, speed of deployment, the possibility of remote administration, and automatic installation. As a result, it was found that the user requirements for software from different types of firms are almost the same and are reduced to the convenience of installing and working in software [7]. Notably, the literature has provided various data about the financial impact of the introduction of electronic medical records, which saves 20.0-63.4% of the working time of medical staff, with a calculation of economic efficiency of US$5530 per physician annually (United States) or in the range of 12-20% (Norway) [8,9].

The purpose of the study was to assess the organizational effectiveness of the introduction of a healthcare information system (HIS), including electronic medical records and databases, in healthcare in Kazakhstan using a combination of 2 methods: the expert assessment method and strengths, weaknesses, opportunities, threats (SWOT) analysis.

METHODS

The study involved 40 experts drawn from specialists in primary healthcare in Aktobe, including 15 representatives of administrative and managerial personnel (chief doctors and their deputies, heads of medical statistics offices and organizational and methodological offices, and internal audit services), and 25 general practitioners. All participants (doctors working in 4 polyclinics in Aktobe), agreed to be experts and were included (100% response rate). The inclusion criteria were: staff position (chief physicians and their deputies, heads of medical statistics offices and organizational and methodological offices, internal audit services, and general practitioners), age (at least 30 years and no more than 60 years), and experience in practical healthcare for at least 5 years. A brief description of the socio-demographic data of the participants (compliance with the inclusion criteria) is presented in the maps of the expert survey in the section “1. Information about experts.” The exclusion criteria were a position other than the above, age (less than 30 years or more than 60 years), and work experience in practical healthcare for less than 5 years. None of the candidates for experts were excluded during the study.

When coordinating expert opinions and developing a final examination, it is important to observe the following requirements: consistency and mutual consistency of the examination parameters, validity and consistency of conclusions, and completeness of the solution of the task. Compliance with these requirements ensures high-quality expertise, provided, of course, the appropriate competence of experts. In general, the reliability of expertise is determined by 2 main factors: first, the qualifications of experts and their ability to solve problems of this level of complexity, and second, the qualifications of the research team and its ability to select competent experts and acquire and generate expert knowledge about the problem. The study followed these principles of choosing experts. The classical technique of SWOT analysis is quite fully described by Dias et al. [10] and Maysak [11]. SWOT analysis involves the division of factors describing the object of research into 4 categories: strengths, weaknesses, opportunities, threats. Strengths and weaknesses are factors of the internal environment of the object under study (that is, what the object of study is capable of influencing), and opportunities and threats are factors of the external environment (that is, what can affect the object from the outside and at the same time cannot be directly controlled by the object). When conducting a SWOT analysis of the features of the implementation of the HIS, the following stages were conducted: (1) Identification of factors describing the current state of the implementation of HIS in the polyclinics of Aktobe (expert survey) and (2) Statistical assessment and ranking of identified factors by importance (weights) and truth (probability).

The method of expert assessment included a 2-stage survey using the method of a correspondence questionnaire (i.e., 2 surveys of the same experts on the same questionnaire, but with different types of questions: open and closed). The 2-stage survey was conducted twice using two questionnaires. The first questionnaire contained the classic 4 questions of SWOT analysis about the features of the introduction of HIS in the polyclinics of Aktobe. The second questionnaire included questions on indicators for assessing the organizational effectiveness of the implementation of HIS in the polyclinics of Aktobe. The first survey was conducted using a structured questionnaire that also contained open-ended questions. Before the survey, a group consultation of experts working at 1 medical organizations (polyclinic) was conducted, and a total of 4 consultations were held at 4 polyclinics in Aktobe. The purpose of the study was presented at the consultation, the participants were given explanations about the rules of the survey (anonymity, honesty, completeness of answers, use of their own practical experience), and the need for a 2-step survey was substantiated. Then the participants were given questionnaires, which they filled out within 1 hour. After the first survey, the questionnaires were processed by a researcher who brought together all the opinions of experts, including individual answers, and structured them.

A week later, the second round of the survey was conducted, but this time using a structured questionnaire with closed-type questions. After a brief explanation of the essence of the second round, the experts filled out questionnaires within 1 hour, taking time to think about the weight values for the factors presented. The first questionnaire was based on 4 basic questions of classical SWOT analysis. SWOT focuses on the strengths and weaknesses of an organization, and opportunities and threats from the external environment. “S” and “W” refer to the state of an organization, and “O” and “T” refer to the external environment of the organization. The second questionnaire included questions on indicators for assessing the organizational effectiveness of the implementation of HIS in the polyclinics of Aktobe. The main blocks of work performed by medical staff and the tasks of this block were independently identified. The experts then gave their assessment in both qualitative and quantitative terms. The following components of work in the polyclinic were assessed: Registration and movement of medical documentation, statistical reporting, and results of processes (research, consultations); Administrative control; Financial costs (production costs); Financial costs (non-production costs); Employees’ working time costs; Information security; and Patient satisfaction.

To eliminate statistical errors, a quantitative expert assessment of individual SWOT factors was carried out after filtering the total number of identified factors to reduce their number when compiling the final version of the SWOT analysis matrix. Expert opinions were coordinated according to the average scores, with the opinions of experts brought to a common denominator. A simple average score was calculated. The results of the expert SWOT survey were statistically calculated based on a weighted score. Using a computer program for SWOT analysis, an average weight-probability indicator was calculated for each factor according to the following equation (1) [11]:

(1)

F=∑i=1npiwin,

where: pi the truth (probability) of the factor, wi the importance (weight) of the factor; i ϵ [1..n] – the number of the expert; F ϵ [0, 1].

The extended SWOT matrix 1.0 software suite was used to calculate the SWOT analysis indicators and visualize the results. The final matrix contains the sums of weighted expert assessments. Next, a rank assessment of the identified factors was carried out according to their degree of importance and the final results of the analysis were drawn up. Descriptive results are presented in the form of averages and standard deviations. The number of expert responses to each question is presented as a percentage (n=40). The normality of the data distribution was checked using the Shapiro-Wilk test. The sample size was 28.1% of all doctors working in 4 polyclinics in Aktobe, which justifies the statistical reliability of the results at the level of 95% or more by the Student-test (t≥3). The statistical significance of the results was estimated, with a coefficient of p-value <0.05 considered statistically significant.

Ethics Statement

The project was approved by the Bioethical Committee of the West Kazakhstan Medical University (Aktobe, Kazakhstan). Participants were provided with a detailed explanation of the study and informed consent was obtained.

RESULTS

The average age of the participants was 42.6±2.3 years (range, 31-60), and 32 of them were women (80%). The average period of practical work experience in the healthcare system was 18.8±3.4 years. Notably, there was no division within the group by age or experience. At the first stage of the analysis, experts identified the factors characterizing the features of the introduction of HIS in the polyclinics of Aktobe (Table 1).

In the second stage of the analysis, the identified factors were assessed and ranked by importance (weight) and truth (probability). The results are presented in Table 2.

The expert assessment showed that the majority of experts (90.0±6.8%) were confident that it is necessary to improve the computerization of the polyclinics. The majority of experts (64.7±6.5%) believe that outpatient medical organizations need further technical equipment. This is consistent with data from cross-sectional studies in 30 countries, including Russia and Kazakhstan [9,12,13]. More than half of the experts (63.0±6.6%) were confident that remote counseling of patients in health schools can improve the quality of medical care to the greatest extent. In addition, 57.4±6.7% of experts suggested that the introduction of IT in the organization of health schools would also improve the quality of medical care. Most experts were sure that both patients (61.1±6.6%) and doctors (68.5±6.3%) were ready for remote consultations. Similar results were reported in studies by Stolyar et al. [14] and Thiyagarajan et al. [15].

From a large number of approaches to assessing the effectiveness of the implementation of an information system in organizations, an approach based on a clear statement of measurable goals before starting a project and monitoring their achievement based on its results was chosen. In most studies, measurable indicators are assessed in the form of economic indicators, the researchers did not consider them, since the decision on the introduction of information systems, and accordingly, the economic justification of an IT project in healthcare is made by the governing bodies (the Ministry of Health of Kazakhstan) [16]. The qualitative effects of the introduction of HIS in general for all industries/companies were analyzed. In the scientific literature, qualitative indicators of the effectiveness of IT implementation are not fully presented, but clear recommendations are given that indicators should be developed considering the specifics of each enterprise. For example, the following indicators may be used: increases in the investment attractiveness of the enterprise, improvements in organizational discipline, establishment of a unified information environment, and scalability.

To determine the effectiveness of the implemented HIS for a specific medical organization (of a certain type), it is necessary to define a system of indicators of the effectiveness of automated business processes and conduct their predictive assessment. A system of indicators is developed individually for each business process. Qualitative characteristics of the business process were used as indicators, which were further assessed by experts in scores (Table 3).

As a result of an expert assessment based on a point ranking, the following qualitative indicators of the effectiveness of the implementation of HIS in outpatient clinics were identified: in the top place, improvement of administrative control (36.3%), followed in descending order by registration and movement of medical documentation (33.5%), statistical reporting and the results of processes (research, consultations) (12.5%), and the cost of working time of employees (6.2%). At the same time, all experts noted that there was no reduction in financial costs (non-production costs), namely in terms of the costs of copying and delivering information in paper form, technical equipment, and paper. Patient satisfaction also improved relatively slightly. Spinazze et al. [17] and Patel et al. [18] highlighted other indicators, such as improved administrative control and reduced time for the effectiveness of the laboratory service. Next, the experts assessed the technical indicators of the implementation of HIS in outpatient clinics (Table 4). The most significant (but negative) indicator was the need for data duplication (weighting factor: -17.9%), followed in descending order by the speed of data entry and analysis and the speed of program deployment (14.3% each), the ease of use of the program and the availability of computers (10.7% each), and IT integration (interconnection of several programs), user-friendliness of the a (“hang-ups” of the program) 7.1% each.

DISCUSSION

Similar results were obtained in studies Corley [19] and Beckmann et al. [20].

According to the expert survey, the goals and results of the implementation of information systems in outpatient clinics were identified and clarified (Table 5).

In the third stage of the analysis, the experts developed proposals to improve the functioning of the HIS in the polyclinic: (1) Measures to improve the HIS: Inclusion of an “Immunization” section in the “Polyclinic” HIS; Inclusion in the “Polyclinic” HIS of the possibility of attaching scanned documents (hospital discharge, expert opinions, analyses); Exclusion of duplicate data entry.

(2) Measures to improve the organization of work at the HIS in outpatient clinics: Organization of remote consultation of patients via the internet, the inclusion of a special section in the polyclinic’s web portal; Development of a mobile application for dispensary patients, including those with arterial hypertension.

(3) Measures to improve the skills of medical workers: Further development of computer literacy and the use of Internet resources for professional purposes; Further improvement of skills in medical ethics and deontology, in connection with the use of Internet resources; Development of accessible technologies of remote forms of advanced training of medical workers; Providing medical workers with up-to-date professional information (regulatory documents, pharmacopeias, medical news, scientific articles) on a regular basis, and information exchange within the industry.

There are a large number of different approaches, methods, and techniques to analyze and assess the effectiveness of information technologies using a variety of heterogeneous criteria and conceptual approaches. As evidenced by practice, most of the methods are cumbersome and difficult to apply in practice, and the degree of their objectivity and the cost of their application varies significantly. In the scientific literature, the conventional analysis of the effectiveness of IT applications is most often used, which is assessed in financial terms; however, some of the effects, especially in healthcare, cannot be financially evaluated, although in the future IT applications may affect the financial results of the organization. Therefore, intangible benefits are often determined first, and then quantitative savings are calculated for each of the types of benefits (for example, cost comparisons before and after the introduction of information technology, and direct calculations of the cost of medicines). At the same time, intangible benefits are determined by timing the working hours of medical personnel (patients or managers) and interviewing experts.

Currently, more than 22 healthcare information systems have been implemented in all medical organizations in Kazakhstan, but this innovation has not solved current health problems, such as long waits for appointments for hospitalization or for appointments with public healthcare specialists, poor quality of medical services, and dissatisfaction with the work of medical staff. Thus, the SWOT analysis in this study showed that outpatient medical organizations are currently equipped with computer equipment and software that provide access to medical information, which allow for more timely measures on various types of activities of a medical organization, and there is also a need for continuous training, effective scientific research, active professional interaction with medical professionals to optimize the use of information systems. The introduction of remote counseling in preventive work in polyclinics increases the patient’s ability to receive the necessary information from the district doctor via the internet to their questions on methods of prevention of chronic non-communicable diseases and risk factors for their development.

The narrow focus of this study made it possible to study the issue of the relevance of digitalization in the healthcare sector in considerable detail; however, conducting studies with a large number of participants and raising topics related to the main topic of the work would help to make the article not so limited. One of the future research directions on the topics raised herein could be the evaluation of the implementation of HISs throughout the country.

Notes

CONFLICT OF INTEREST

The authors have no conflicts of interest associated with the material presented in this paper.

FUNDING

This work was carried out with the financial support of grant No. IRN09562783 “Improvement of geriatric and geriatric care in the Republic of Kazakhstan”.

ACKNOWLEDGEMENTS

Notes

AUTHOR CONTRIBUTIONS

Conceptualization: Yermukhanova L. Data curation: Yermukhanova L, Buribayeva Z. Formal analysis: Buribayeva Z. Funding acquisition: Yermukhanova L. Methodology: Abdikadirova I, Kurganbekova M. Project administration: Abdikadirova I, Buribayeva Z. Visualization: Tursynbekova A, Kurganbekova M. Writing – original draft: Tursynbekova A, Kurganbekova M. Writing – review & editing: Yermukhanova L, Abdikadirova I, Buribayeva Z.

REFERENCES

• 1. Order of the Ministry of Health of the Republic of Kazakhstan. On approval of the concept for the development of e-health in the Republic of Kazakhstan for 2013-2020; [cited 2022 Aug 18]. Available from: https://cutt.ly/iEzdipm (Russian)
• 2. Schleicher DJ, Baumann HM, Sullivan DW, Yim J. Evaluating the effectiveness of performance management: a 30-year integrative conceptual review. J Appl Psychol 2019;104(7):851-887ArticlePubMed
• 3. Lawrie G, Cobbold I. Third-generation balanced scorecard: evolution of an effective strategic control tool. Int J Product Perform Manag 2004;53(7):611-623Article
• 4. Nielsen ME, Skaarup S. IT portfolio management as a framework for managing technical debt: theoretical framework applied on a case study. 14th International Conference on Theory and Practice of Electronic Governance (ICEGOV 2021), October 06-08, 2021, Athens, Greece. New York: Association for Computing Machinery; 2021. 89-96
• 5. Colicchio TK, Facelli JC, Del Fiol G, Scammon DL, Bowes WA 3rd, Narus SP. Health information technology adoption: understanding research protocols and outcome measurements for IT interventions in health care. J Biomed Inform 2016;63: 33-44ArticlePubMed
• 6. Temnogrudova EA. Evaluation criteria of effectiveness of information techno logy in the organization. Models Syst Netw Econ Technol Nat Soc 2014;3(11):80-83. (Russian)
• 7. Kano N, Seraku N, Takahashi F, Ichi Tsuji S. Attractive quality and must-be quality. J Jpn Soc Qual Control 1984;14(2):147-156. (Japanese)
• 8. Wang SJ, Middleton B, Prosser LA, Bardon CG, Spurr CD, Carchidi PJ, et al. A cost-benefit analysis of electronic medical records in primary care. Am J Med 2003;114(5):397-403ArticlePubMed
• 9. Stobbe EJ, Groenewegen PP, Schäfer W. Job satisfaction of general practitioners: a cross-sectional survey in 34 countries. Hum Resour Health 2021;19(1):57ArticlePubMedPMCPDF
• 10. Dias C, Santos M, Portela F. A SWOT analysis of big data in healthcare. In: Proceedings of the 6th International Conference on Information and Communication Technologies for Ageing Well and e-Health (ICT4AWE 2020). Setúbal: Science and Technology Publications; 2020. p. 256-263
• 11. Maysak OS. SWOT analysis: the difficulty of searching for links between factors. Casp J Manag High Technol 2013;1(21):151-157. (Russian)
• 12. Zulin YV, Bortsov VA. Main directions of organizational improvement of ambulant and polyclinic care for population; [cited 2022 Aug 8]. Available from: https://cyberleninka.ru/article/n/osnovnye-napravleniya-sovershenstvovaniya-organizatsii-ambulatorno-poliklinicheskoy-pomoschi-naseleniyu
• 13. Imanova ZhA, Mukhanova GT, Sakhanova LKh, Imambaev NI, Naurzalieva AD. Creation of an effective model of providing medical care to the rural population, including drug provision, taking into account international experience. Nur-Sultan: Republican Center for Health Development; 2020. p. 1-44 (Russian)
• 14. Stolyar V, Lyapunova T, Amcheslavskaya M, Lukianova E, Shimkevich E. Methodology of teaching on telemedicine technologies for medical students. Eurasia Proc Educ Soc Sci 2018;10: 254-257
• 15. Thiyagarajan A, Grant C, Griffiths F, Atherton H. Exploring patients’ and clinicians’ experiences of video consultations in primary care: a systematic scoping review. BJGP Open 2020;4(1):bjgpopen20X101020ArticlePubMedPMC
• 16. Order of the Acting Minister of Health of the Republic of Kazakhstan. On approval of forms of primary medical documentation for healthcare organizations; [cited 2022 Aug 18]. Available from: https://adilet.zan.kz/rus/docs/V1000006697 (Russian)
• 17. Spinazze P, Aardoom J, Chavannes N, Kasteleyn M. The computer will see you now: overcoming barriers to adoption of computer-assisted history taking (CAHT) in primary care. J Med Internet Res 2021;23(2):e19306ArticlePubMedPMC
• 18. Patel S, Boulton KA, Redoblado-Hodge MA, Papanicolaou A, Barnett D, Bennett B, et al. The acceptability and efficacy of electronic data collection in a hospital neurodevelopmental clinic: pilot questionnaire study. JMIR Form Res 2021;5(1):e18214ArticlePubMedPMC
• 19. Corley ST. Electronic prescribing: a review of costs and benefits. Top Health Inf Manage 2003;24(1):29-38PubMed
• 20. Beckmann M, Dittmer K, Jaschke J, Karbach U, Köberlein-Neu J, Nocon M, et al. Electronic patient record and its effects on social aspects of interprofessional collaboration and clinical workflows in hospitals (eCoCo): a mixed methods study protocol. BMC Health Serv Res 2021;21(1):377ArticlePubMedPMCPDF

Figure & Data

References

Citations

Citations to this article as recorded by  

• Analysis of European accounting and auditing firms: do they have different business viability?
  Vera Gelashvili, Alba Gómez-Ortega, Almudena Macías-Guillén, María Luisa Delgado Jalón
  The Journal of Risk Finance.2024;[Epub]     CrossRef