INTRODUCTION

Vaccinations have contributed to decreasing serious outbreaks by preventing incidence and propagation of contagious diseases in advance. The vaccination program is a cost-effective enterprise that provides vaccines for preventing the outbreak and propagation of vaccine-preventable diseases. Implementing a high vaccination coverage rate and the effect of the vaccine on the disease will determine whether or not a vaccination program can successfully prevent an outbreak or the propagation of the disease [1]. Recently, many developed and developing countries have pushed forward with extending the coverage of required vaccinations for every resident their jurisdictions, and many countries are making efforts to eliminate the geographic and economic limits to vaccination among low and vulnerable classes. In Korea, phased extension policy was chosen by government as the coverage plan for the required national vaccinations. Thus, the private medical institution participation method of the National Expanded Program on Immunization had been implemented since 2005 [2]. Daegu Metropolitan City and Gunpo City participated in the pilot project that enforced vaccinations for 13 infectious diseases. The objects were children 0 to12 years old and the project was performed at private medical institutions; the objects were reimbursed for the entire cost of their individual inoculations [2]. This developed into the enforcement of the National Immunization Program (NIP), since 2009, in which a child can receive the part of their inoculation cost at most private medical institutions in Korea [3]. If a child 12 years old and younger is vaccinated at a designated private medical institution, the cost of the vaccination is covered by the government on the condition that the patient receives vaccinations for 8 of the required vaccinations including 22 administrations. Implementing such programs for increasing vaccination rates has been one of the elements that enabled us to prevent contagious diseases; the Korean government has been making such policy-related efforts for many years.

However, in order to ensure success for the vaccination program, the efforts not only toward increasing vaccination rates, but also toward maintaining the quality of vaccines are needed. Due to the fact that most vaccines maintain their potency for a short amount of time at room temperature, the cold chain is the core of the system used for managing the quality and safety of vaccines. In addition, the principles related to management of vaccines such as installation of thermometers, maintenance of appropriate temperatures, recording temperatures, location of the vaccines inside a refrigerator, defrosting a frozen vaccine, as well as using the refrigerator exclusively for vaccine storage must be followed in order to finally administer the vaccine at full titer. For example, in one localized outbreak of measles in the U.S. during the 1970s, illness was associated with inoculation at a medical institution where the vaccines were stored on a refrigerator door shelf, not in the refrigerator's center [1]. In Australia, it was claimed that incorrect storage of vaccines could have contributed to the resurgence of pertussis [1]. Likewise, the vaccination program can be successful in preventing the occurrence and spread of illness only if the effect of a vaccine itself can be maintained through biological preparation; the storage and management of the vaccine are important, but the regulations are incomplete, and the distribution enterprise and healthcare institution management in the field is even more vulnerable [4]. Moreover, there is almost no research that has intensively discussed the actual state of vaccine storage and management, and standardized guidelines along with systematic education programs to address these issues are rare. This study focused on inspecting actual vaccine storage status and awareness at private medical institutions, and compared them before and after education was provided.

METHODS

I. Study Subjects

Among the private medical institutions located in 4 regions (Nam-county in Pohang-city, Yeongju-city, Sangju-city, and Mungyeong-city) in the Gyeonsang buk-province, we randomly selected 10 medical institutions (2 hospitals and 8 clinics that had been entrusted with the NIP and that had carried out infant/children vaccination in August 2010) from each region. Out of a total of 40 private medical institutions, we investigated 39 that had agreed in advance over the phone to our visit. We requested a visit to those medical institutions, but we did not speak about the details such as whether or not the medical institutions possess a refrigerator exclusively for vaccine storage, or asking for records of the refrigerator temperature.

The investigation was performed over two time periods (primary and secondary inspection). The person in charge of the vaccinations and a private medical institution supervisor of each public health center in the jurisdiction of the private medical institutions were visited. Using the checklist, we observed the actual status of vaccine storage and handling, and investigated the awareness of the person in charge of vaccine management using the questionnaire. The questionnaire was performed under the principle that stated that the person in charge of vaccine management in each private medical institution would be the subject of the research, but in case there was not a designated person in charge, the questionnaire was performed based on the person who conducted the most work, on average, relating to vaccine management.

The checklist was prepared for grasping the actual state of vaccine storage and handling, and was self-produced by referring to existing studies and foreign guidelines [1,5,6]. The checklist content includes the following: whether or not there was a designated person in charge of vaccine management, whether or not the person had undergone vaccine management training, whether or not the temperature for vaccine storage was monitored and recorded, whether or not the appropriate temperature (2℃ to 8℃) was maintained, whether or not the expiration date was heeded, and whether or not equipment of refrigerator reserved for vaccine storage and emergency provision. A person in charge of vaccination and a private medical institution supervisor of a public health center inspected the checklist items.

The questionnaire for determining the awareness of proper vaccine storage and management procedures was self-produced by compiling from previous studies and foreign guidelines [1,5,6]. After inspecting the checklist, the questionnaire was administered by person in charge of the public health center vaccinations. Using Likert scale for the questionnaire answers, out of five answer choices ('strongly agree', 'agree', 'neither agree nor disagree', 'disagree', 'strongly disagree'), we let the subjects provide answers to the questions according to their knowledge regardless of the present storage practices.

After the questionnaire was completed, the people in charge of vaccinations at each public health center who were also well-informed of knowledge related to vaccine storage supervised the one-on-one education of the questionnaire subjects. The education manuals used were leaflets that were self-produced in advance.

The content of the education manuals were as follows: 1)When receiving delivery of vaccine, to maintain the cold chain, check whether or not the packaging has been destroyed, and check the expiration date. 2) When storing a vaccine, the appropriate temperature is 2℃ to 8℃. 3) Record the temperature of the refrigerator every hour. 4) Preserve a daily record of the temperature over the previous two years. 5) Install a refrigerator thermometer in the center of the refrigerator. 6) Do not store vaccines in areas where a change in temperature occurs easily, such as a refrigerator door shelf or the front of a refrigerator. 7) Do not keep other kinds of drugs or food items in the vaccine refrigerator. 8) Store vaccines according to their type for the purpose of easy distinction. 9) Expired vaccines need to be discarded immediately. 10) The emergency power supply, in case of blackout, needs to be checked once a week. 11) The opening time of multi-dose vials needs to be recorded. 12) In case a vaccine is opened, do not store it again and do not re-use it the next day.

The secondary inspection was performed using an identical questionnaire and identical checklist by revisiting the same medical institution 4 weeks after the day of the primary inspection. The subjects of the secondary inspection were the same persons as those in the primary inspection.

II. Statistical Analysis

In order to compare the actual status of vaccine storage and handling, the appropriate procedure related to each item was treated as 1 point. In the questionnaire, where the positive answers were the correct answers, 'strongly agree', and 'agree' were treated as 1 point, and 'neither agree nor disagree', 'disagree' and 'strongly disagree' as 0 points. Where the negative answers were correct, 'disagree' and 'strongly disagree' were treated as 1 point, and 'neither agree nor disagree', 'agree', and 'strongly agree' as 0 points. In order to determine if each item of the questionnaire and the checklist had any distinction according to the type of medical institution, we used the chi-squared test. We added all the scores for each item, and we compared the average score of the actual status and awareness by type of medical institution through one-way ANOVA.

We compared whether or not there was a difference in the actual procedures followed before and after education and inspection, according to the items, using the Mcnemar's chi-squared test. We compared the total score of the primary and secondary inspections using the paired t-test. We did a comparison of the awareness questionnaire results, according to the items, using the Wilcoxon's signed-rank test. And the total scores of the primary and secondary inspection were compared using the paired t-test. All statistical analyses were performed using SPSS version 17.1 (SPSS Inc., Chicago, IL, USA).

RESULTS

I. General Characteristics

We report the general characteristics of the questionnaire subjects and the relevant private medical institution in Table 1. According to the analysis of the research subjects, 18 (46.2%) were male, 21 (53.8%) were female, and the majority were in their 30s and 40s (16 people in each 10-year age range, 41.0%). The order according to duty was as follows: 20 physicians (51.3%), 12 nurses (30.8%), 3 nurse aides (7.7%), 2 pharmacists (5.1%), and 2 members of office staff (5.1%). For work experience at a medical institution, 37 people had worked for more than 5 years, and the majority had worked for more than 11 years (25 institutions, 64.1%). Among the 39 private medical institutions investigated, there were 8 hospitals (20.5%), 12 pediatric clinics (30.8%), and 19 other clinics (miscellaneous clinics, 48.7%). As for the average number of vaccinated people, in one week, the majority of medical institutions (25 institutions, 64.1%) were vaccinating less than 20 people, and 14 institutions vaccinated less than 10 people (35.9%). Fifteen institutions had more than 13 kinds of vaccines (38.5%).

II. Vaccine storage and Management Practice

Table 2 shows the checklist items for the vaccine storage and management practice inspection, as well as the first set of inspection results. Out of the 39 private medical institutions, only 3 (7.7%) had refrigerators exclusively for storing vaccines, and they were all hospital-level medical institutions. Most of the medical institutions were maintaining the appropriate temperature of the refrigerator for storing vaccines. All 12 pediatric clinics had maintained an acceptable temperature but 2 miscellaneous clinics and 1 hospital maintained the refrigerator temperature deviated from the acceptable range. There was no significant difference among the three types of institutions. In 2 medical institutions (5.1%), vaccines that needed to be in the freezer were stored inside the refrigerator, and both of them were pediatric clinics. A total of 10 (25.6%) medical institutions stored multi-dose vials without recording the opening dates and as many as 6 miscellaneous clinics (31.6%) were shown but there were no significant differences among the three types of institutions. Other than the aforementioned items, some of items that showed significant differences according to the type of medical institution were whether or not the medical institutions had a refrigerator exclusively for vaccine storage, whether or not the institution had emergency power supplies in case of a blackout, and whether or not there were extra refrigerators in case of an emergency. There were no pediatric clinics that with an emergency power supply in case of a blackout, but extra refrigerators were equipped the most in pediatric clinics (91.7%). Overall, item number 11, whether or not the medical institution stored expired vaccines, was adhered to the most as none of the 39 institutions were storing expired vaccine.

The score for the checklist of the primary inspection was 9.74 ± 0.39 (SE) out of 15 possible total points. The comparison results, according to the kind of medical institution, were in the order of pediatric clinics 10.58 ± 0.39, hospitals 9.62 ± 0.62, and miscellaneous clinics 9.26 ± 0.71. There were no significant differences in checklist scores according to the type of medical institution (Figure 1).

III. Awareness Investigation

The questionnaire questions and results for the awareness investigation before education are shown in Table 3. The question that was answered incorrectly the most was that using even a general refrigerator is appropriate for storing vaccines as long as the temperature is maintained properly (item 4), and only 6 people (15.4%) out of all subjects answered 'strongly disagree' or 'disagree'. There was a significant difference according to the type of medical institution, and in particular, all the subjects of the pediatric clinics had the wrong answer for that question. Items answered correctly included 'a refrigerator for vaccine storage only is needed' (item 2), 'vaccines should be stored in a refrigerator designated only for vaccine storage' (item 3), which had 26 people (66.7%) and 25 people (64.1%), respectively. Though there was no significant difference according to the types of medical institutions there was a low awareness demonstrated in pediatric clinics. There were 3 people (7.7%), all from pediatric clinics, who knew that freeze and thaw cycles are allowed with frozen vaccines (item 9), and the difference according to types of medical institutions was significant. The question with the highest number of subjects that showed proper awareness was that multi-dose vials of vaccines should be discarded within a certain period after opening (item 8).

The average awareness score of the primary inspection was 10.48 ± 0.34 (SE). Based on the results according to the type of medical institution, the order was as follows: hospitals 11.75 ± 0.86, miscellaneous clinics 10.36 ± 0.49, and pediatric clinics 9.83 ± 0.50. The resulting value was highest overall for the hospitals but there was no significant difference (Figure 2).

IV. Comparison of Before and After Education

We compared the actual status of vaccine storage between the primary and secondary inspection. Medical institutions that record the refrigerator temperature (item 4) increased significantly from 27 institutions (69.2%) to 34 institutions (87.2%). The number of medical institutions that stored vaccines in the center of the refrigerator (item 8) increased significantly from 17 institutions (43.6%) to 26 institutions (66.7%), and institutions that stored vaccines with other kinds of medications or food significantly decreased from 14 institutions (35.9%) to 8 institutions (20.5%) (Table 2). The average secondary inspection score of the checklist for all of the private medical institutions together was 10.58 ± 0.34 (SE), and it significantly increased by 0.84 points compared to the primary inspection (p<0.001) (Figure 1). For each type of medical institution considered separately, the score after education for hospitals increased by 0.75 (p=0.320), pediatric clinics by 0.75 (p=0.032), and miscellaneous clinics by 0.95 (p=0.002) (Figure 1).

Awareness after education, compared to before the education, significantly improved in items 5-7, which were about the completion of temperature records, exclusive storage of vaccines, and selective storage of vaccines according to the type and expiration date, as well as items 10-14 which were light-proof storage, installation of a thermometer in the center of the refrigerator, storing the vaccines in the center of a refrigerator, installation of an independent generator, and educating people in charge of vaccines. The awareness score of all the medical institutions after education was 11.51 ± 0.28 (SE), and it increased by 1.03 points compared to before the education (p<0.001) (Figure 2). According to the types of medical institutions, hospitals increased by 0.75 points (p=0.197), pediatric clinics by 1.25 points (p=0.006), and miscellaneous clinics by 1.00 point (p=0.005) (Figure 2).

DISCUSSION

If a group of people has a high immunity level, even with some susceptible people in the group, there is a small possibility that an outbreak can occurs. In particular, if a human is the only host organism, increasing the level of immunity of a group can eradicate an infectious disease [7]. Using this method, smallpox has been the only disease humankind has completely eradicated up to the present time, but the number of eradicated diseases can be increased through vaccination programs. Diseases such as polio and diphtheria are at the level of being almost entirely eradicated [2]. However, until the present time, the focus has been on coverage rates of vaccines, and the problems related to the quality of handling after the production of a vaccine have received little attention. Then, Lerman and Gold [8] in 1971 and Krugman et al. [9] in 1974 reported that one plausible factor that resulted in outbreak of measles in children with a vaccination history was improper storage of vaccines and thus the effect of vaccines could have been reduced. Therefore, problems related to storage and management of vaccines captured people's attention. However, reviewing the following studies, recognition related to the importance of vaccine storage has not been well established yet, and the actual post-production handling conditions have been diverse [5,10-15]. Steinmetz et al. [10] investigated the actual conditions for storing live measles, mumps, and rubella virus vaccines. In 1981, 60% of the investigated medical institutions had the refrigerator temperature at higher than 8℃, and they reported that only 40% were storing vaccines at the recommended location in the refrigerator. In the research performed by Hunter [11], only 3 medical institutions out of 36 inspected had installed a thermometer inside the refrigerator. In Bishai et al. [12] 22% of refrigerators had an inside temperature that was higher than 8℃, and 46% had stored the vaccines on the refrigerator door shelf. Thakker and Woods [13] stated that out of 40 subjects who answered their questionnaire, only 16 knew the correct vaccine storage method. Yuan et al. [5] investigated both knowledge and actual conditions related to the storage and handling of vaccines from a primary care physician's office, and only 6% of participants provided correct answers to all 4 of the questions, which showed the necessity of awareness improvement. At the same time, they stated that only 10% of the investigated medical institutions had installed thermometers inside their refrigerators, and about one-third of them had temperatures outside the recommended range. Just as mentioned earlier, despite the fact that many studies have reported on the importance and seriousness of vaccine storage and management conditions, the efforts of health personnel including policy-decision-making institutions to remedy the situation have been insufficient.

In Korea, the distribution system for vaccines in private medical institutions is one in which the hospitals and clinics have direct deals with either the manufacturing company or sales company, or are being provided by the wholesale markets; it is very important to properly maintain the cold chain throughout all these processes. The cold chain, also known as the low-temperature distribution system, is a delivery system that maintains a low temperature from the time of production until the time of supply, and is the core principle of management of vaccine quality and safety. Although this study was not able to widen the research to include the whole distribution process including the manufacturing company, sales company, and the wholesale market, the authors apprehended the actual conditions of vaccine storage and handling in private medical institutions located in 4 regions in Gyeongsangbuk-province that performed infant and child vaccinations.

There are two ways in which the titer of the final administrated vaccine can be directly decreased due to an incorrect storage method: one way is to use vaccines that were not stored at the appropriate temperature, and the other is to use vaccines whose expiration date has passed [6,9]. The latter can include using opened multi-dose vials of vaccine that have been exceeded a certain acceptable time period for use, after they have been opened. The questionnaire items that most directly reflect the two abovementioned ways are 6, 7, 11, and 12. The checklist for actual status of item 6 is the measurement of direct storage temperature, and item 7 showed that vaccines were stored at a wrong temperature. Items 11 and 12 reflect whether it is possible to use vaccines anytime from slightly to extremely past their expiration date. As a result of the primary investigation, 3 medical institutions were not maintaining the refrigerator temperature at 2℃ to 8℃, and there were even as many as 2 institutions that had stored vaccines in the refrigerator that were supposed to be stored in the freezer. The former institutions were all hospitals or miscellaneous clinics, whereas, the latter were all pediatric clinics. However, we cannot determine whether a certain type of medical institution had an actual serious problem related to vaccine storage based only on the results mentioned above, because the temperature of the refrigerators were measured only once, and vaccines that have to be kept inside the freezer like varicella-containing vaccines are administrated frequently at pediatric clinics. In addition, this is because there was a practical limit on the number of investigation subjects. None of the medical institutions that participated in the investigation that stored expired vaccines, but 25.6% of the institutions did not record the opening date of multi-dose vials of vaccine. Out of those 25.6% institutions, there were relatively more miscellaneous clinics, but the difference among types of institutions was not significant. The important thing to keep in mind is that administrating vaccines without abiding by items 6, 7, 11, and 12 can possibly lead to failure of formation of the antibody.

Among the private medical institutions investigated, 3 of them were furnished with refrigerators exclusive for vaccine storage, but most of them were storing vaccine in typical refrigerators. For storing vaccines, stand-alone freezers and refrigerators without freezers are preferred because maintaining the temperature is easier than with a combination unit [6]. Even if the equipment has a refrigerator and freezer combined, each compartment must have an exterior door and temperature-maintaining equipment. The type of refrigerator that should not be used for storing vaccine is a dormitory-style refrigerator. This type of refrigerator has access to the refrigerator and freezer at the same time without a separate door, which forms one small unit. The cooling coil is normally located in the freezer section of this type of refrigerator, and the temperature sensor is not sensitive enough to fulfill its intended function. In particular, once it decreases the temperature, many times the temperature of the refrigerator falls below 0℃. Even if the vaccine is stored for a short period of time, it is necessary to use a refrigerator that is appropriate for storing vaccine to prevent the vaccine from being exposed to a temperature that is either too high or too low, thus, preventing the spoiling of the vaccine [6].

After the primary inspection, it was found that only 7.7% of the medical institutions were storing only vaccines in the refrigerator. It is unacceptable to store other medical products or food in a place where vaccines are stored, because other drugs of foods can pollute vaccines. In addition, there would be more frequent opening and closing of the door, which could make it difficult to maintain the appropriate temperature for the vaccines [6]. All the better, maintaining the appropriate temperature is very important, and thus regular checking of the temperature is very important. However, the principle of keeping the vaccine in the center of a refrigerator, the location where the temperature is rather consistent, was adhered to by less than half of all the institutions inspected (43.6%). Refrigerators without a thermometer were found in as many as 12.8% of the institutions, and 69.2% of institutions recorded the storage temperature regularly. In addition, only 15.4% had an available emergency power supply in case of a blackout. From these facts, it was found that many medical institutions were exposing vaccine to unstable temperatures and that caution about this was lacking [12]. Not only can this decrease the titer of vaccine, but it can also lead to discarding vaccine and re-vaccination due to failure of antibody formation, which would lead to decrease in the credibility of the vaccine efficacy to the patients.

The average score from the checklist was 64.9%, which was a rather low in percentage, so we discovered that the overall actual condition needs to be ameliorated. Although there was not a significant difference according to the type of institution, pediatric clinics had a slightly higher score than the other institutions. Because pediatric clinics provide many patients with vaccinations, there was a high possibility that the storage conditions would have been good [1].

The average score from the awareness questionnaire was 74.9%, and the figure was relatively higher than the storage condition average score. However, based on the point of view that one needs to have a correct understanding for every item in order to handle vaccines properly, this awareness score is not satisfactory. Yuan et al. [5], also considered answering all the questions correctly as very important. Pediatric clinics had the lowest awareness score, which was contradictory to the actual conditions of investigation, but there was no significant difference among types of institutions.

When we compared the checklist score and questionnaire score before and after education, we found that the actual conditions of the overall vaccine storage and awareness had improved. All of the 3 types of institutions improved in both actual conditions and awareness; the pediatric clinics and other miscellaneous clinics showed significant differences.

Most of the checklist items were possible to improve mostly through awareness improvement and individual effort, except for whether or not the institution was equipped with refrigerators exclusive for vaccines, and whether or not an emergency power supply was present. After education was provided, we saw that the actual conditions significantly improved for items such as completion of recording vaccine storage temperature, storing the vaccine in the center of the refrigerator, and storing vaccines separately from other drugs or food. Although actual conditions related to vaccine storage such as a change in behavior of the vaccine storage manager could be further improved with a greater effort, the items that were not possible to improve could have been due to the limitations of one-off education. Repeated education for the improvement of behavior needs to be enforced. In addition, if a periodic reminder such as a written document or an electronic document was sent to an institution, along with enforcement of surveillance and feedback, problems related to vaccine storage or management could be largely solved. These items need to be supported with policies, because their fulfillment would significantly decrease without the support of policies.

In awareness questionnaire, compared to before education, after education several items significantly improved, including awareness of filling out the vaccine storage temperature records, storing vaccines separately from other drugs or food, storing vaccines according to type and expiration date, storage in light-proof conditions, installation of a thermometer in the center of the refrigerator, not storing vaccines on the refrigerator door shelf, the necessity for having an independent generator available in preparation for a blackout, and awareness of the necessity to educate the person in charge of vaccine management. Out of these items in the questionnaire, awareness about storing vaccines according to type and expiration date, and operation of an emergency power supply had significantly improved, but improvement in actual practice was meager. When the subjects were aware of those items, there were a few cases where conditions would have allowed those items to be implemented and practiced, but the reality was that in most of the institutions, resources were insufficient for purchasing a refrigerator exclusively for vaccines or possessing an independent generator. Refrigerators designated specifically for vaccine storage cost 9 million won (about $US 9000) per 780 liters, and realistically speaking, private medical institutions whose quantity of vaccinations is less than 10 cases per week cannot afford vaccine refrigerators. Therefore, policy should not stop at implementing regular inspections and surveillance, but it is imperative that it also include a governmental system that provides resources for private medical institutions to purchase refrigerators for vaccine storage. Most of the problems can be improved by individual effort, but for those problems that cannot be solved too easily, as mentioned earlier, we may assume that it is due to the limits of a one-off education of the person in charge of vaccine management because it may not actually modify behavior itself. Therefore, the results of this research enabled us to confirm that repetitive education, centralized management, and surveillance are important.

Self-produced educational leaflets provided at the same time as one-on-one education included only the basic knowledge required of people in charge of vaccine management. In Korea, the situation is dire enough that we need to produce a standardized manual related to vaccine storage and management. In the U.S., the Centers for Disease Control and Prevention publicly announced their newest standardized guidelines for vaccine storage and handling on their website for users to access easily. On the other hand, in Korea it is difficult to find a standardized guidelines written in Korean except for a couple of sections from the Handbook for safe use of vaccine [16].

There are some limitations to this study. First, the private medical institutions could have had a bias, because we called them on the phone prior to the visit asking for their permission, and although they might not have known the detailed items related to the inspection, they knew we would visit them on the matter of vaccination. Second, we only targeted private medical institutions that were controlled by 4 public health centers in a particular region, which means that the results may not be generalizable to the other regions of Korea. Third, although public health centers administer a large number of vaccinations, we performed the study in collaboration with people in charge of the public health center; thus, we did not include them as study targets, but it is necessary to broaden the range and the subjects in future studies. Fourth, measuring temperatures inside the refrigerators was not done repeatedly; it was only performed once.

Despite these limitations, this study demonstrated that proper awareness and actual practices related to vaccine storage and management are rare in Korea. From a narrow perspective, the goal of this study was to improve the awareness and practices of health care providers in some regions of Gyeongsangbuk-province. By extension, this study is meaningful in the sense that it proposes establishing policies to enforce repeated education, developing standardized manual, and performing periodic management inspection by communicating the results to the people in the public health policy decision-making process throughout the country. In conclusion, vaccination is the most effective and the strongest method ever developed for humans to prevent and eliminate contagious diseases, and a system needs to be established for proper vaccine storage and quality control to increase its efficacy.

Notes

The authors have no conflicts of interest with the material presented in this paper.

This article is available at http://jpmph.org/.

REFERENCES

• 1. Gazmararian JA, Oster NV, Green DC, Schuessler L, Howell K, Davis J, et al. Vaccine storage practices in primary care physician offices: assessment and intervention. Am J Prev Med 2002;23(4):246-253. 12406478ArticlePubMed
• 2. Kim CB, Lee SG, Lee JJ, Jeon SY, Hyun SJ, Lee YK, et al. A comparative study of the administrative behavior of vaccination and satisfaction of physicians according to the national expanded programme on immunization in Korea. Korean J Health Policy Adm 2009;19(3):71-91. (Korean)Article
• 3. Lee KH. Changes in vaccination status and user satisfaction for the full coverage project of the national immunization program (NIP): focus on Gangnam-gu, Seoul [dissertation]. 2010. Seoul: Korea University; (Korean)
• 4. Lee SG. Organization designing for managing efficient immunization and vaccine logistics in KCDC. 2007. Seoul: Korea Center for Disease Control; (Korean)
• 5. Yuan L, Daniels S, Naus M, Brcic B. Vaccine storage and handling. Knowledge and practice in primary care physicians' offices. Can Fam Physician 1995;41: 1169-1176. 7647622PubMedPMC
• 6. Vaccine storage and handling. Centers for Disease Control and Prevention. cited 2012 Mar 12. Available from: http://www.cdc.gov/vaccines/pubs/pinkbook/downloads/vac-storage.pdf
• 7. Ki MR, Kim SD, Bae GR, Lee HJ, Lim HS, Cheon BC, et al. Korean Society for Preventive Medicine. Epidemiology of infectious disease. Preventive medicine and public health. 2011. Seoul: Gyechukmunwhasa; p. 235-315 (Korean)
• 8. Lerman SJ, Gold E. Measles in children previously vaccinated against measles. JAMA 1971;216(8):1311-1314. 4102807ArticlePubMed
• 9. Krugman RD, Meyer BC, Enterline JC, Parkman PD, Witte JJ, Meyer HM Jr. Impotency of live-virus vaccines as a result of improper handling in clinical practice. J Pediatr 1974;85(4):512-514. 4443859ArticlePubMed
• 10. Steinmetz N, Furesz J, Reinhold C, Yarosh W. Storage conditions of live measles, mumps and rubella virus vaccines in Montreal. Can Med Assoc J 1983;128(2):162-163. 6848162PubMedPMC
• 11. Hunter S. Storage of vaccines in general practice. BMJ 1989;299(6700):661-662. 2508856ArticlePubMedPMC
• 12. Bishai DM, Bhatt S, Miller LT, Hayden GF. Vaccine storage practices in pediatric offices. Pediatrics 1992;89(2):193-196. 1734382ArticlePubMedPDF
• 13. Thakker Y, Woods S. Storage of vaccines in the community: weak link in the cold chain? BMJ 1992;304(6829):756-758. 1571683ArticlePubMedPMC
• 14. Cheriyan E. Monitoring the vaccine cold chain. Arch Dis Child 1993;69(5):600-601. 8257186ArticlePubMedPMC
• 15. Woodyard E, Woodyard L, Alto WA. Vaccine storage in the physician's office: a community study. J Am Board Fam Pract 1995;8(2):91-94. 7778494PubMed
• 16. Handbook forsafe use of vaccine. Korea Food and Drug Administration. cited 2012 Mar 20. Available from: http://www.kfda.go.kr/index.kfda?mid=102&pageNo=30&seq=5875&cmd=v

Figure 1

Average score of vaccine storage and handling status and change after education by institution type.

Bars represent the mean±SE, and a perfect score is 15 points; 4 weeks after education, pediatric clinics, other miscellaneous clinics, and the three types in aggregate had increased average scores by 0.75, 0.95, and 0.84 points, respectively, compared with before education (^*p<0.05, ^**p<0.01, ^***p<0.001). There were no significant differences among institution types in scores either before or after education.

Figure 2

Average score of awareness of vaccine storage and change after education by institution type.

Bars represent the mean±SE, and a perfect score is 14 points; 4 weeks after education, pediatric clinics, other miscellaneous clinics, and the three types in aggregate had increased average scores by 1.25, 1.00, and 1.03 points, respectively, compared with before education (^**p<0.01, ^***p<0.001). There were no significant differences among institution types in scores either before or after education.

Table 1.

General characteristics of questionnaire subjects and the private medical institutions where they work (n=39)

Contents	n (%)
Questionnaire subjects
Age (y)
20 - 29	1 (2.6)
30 - 39	16 (41.0)
40 - 49	16 (41.0)
50 - 59	5 (12.8)
≥60	1 (2.6)
Sex
Male	18 (46.2)
Female	21 (53.8)
Duty
Physicians	20 (51.3)
Nurses	12 (30.8)
Nurse’s aides	3 (7.7)
Pharmacists	2 (5.1)
Office staff	2 (7.7)
Years of experience
< 5	2 (5.1)
5 - 10	12 (30.8)
≥11	25 (64.1)
Private medical institutions
Type of institution
Hospitals	8 (20.5)
Pediatric clinics	12 (30.8)
Other miscellaneous clinics	19 (48.7)
No. of vaccination cases (/wk)
<10	14 (35.9)
10-19	11 (28.2)
20 - 29	4 (10.3)
30 - 39	4 (10.3)
40 - 49	2 (5.1)
>50	4 (10.3)
No. of vaccines
1 - 3	4 (10.3)
4 - 6	7 (17.9)
7 - 9	2 (5.1)
10 - 12	11 (28.2)
≥13	15 (38.5)

Table 2.

Vaccine storage and handling status according to itemized checklists by institution types and change after education (n=39)

Vaccine storage and handling practices	Before education	After education	p-value1
1. Has a primary vaccine administrator been assigned?	27 (69.2)	27 (69.2)	>0.99
Hospitals (n=8)	6 (75.0)	5 (62.5)
Pediatric clinics (n=12)	10 (83.3)	11 (91.7)
Other miscellaneous clinics (n=19)	11 (57.9)	11 (57.9)
p-value2	0.36	0.16
2. Has vaccine storage and handling training been provided to personnel who handle or administer vaccines?	29 (74.4)	33 (84.6)	0.29
Hospitals (n=8)	6 (75.0)	6 (75.0)
Pediatric clinics (n=12)	9 (75.0)	10 (83.3)
Other miscellaneous clinics (n=19)	14 (73.7)	17 (89.5)
p-value2	>0.99	0.63
3. Has the refrigerator its own thermometer?	34 (87.2)	36 (92.3)	0.63
Hospitals (n=8)	7 (87.5)	7 (87.5)
Pediatric clinics (n=12)	12 (100)	11 (91.7)
Other miscellaneous clinics (n=19)	15 (78.9)	18 (94.7)
p-value2	0.21	0.78
4. Are storage temperatures being recorded?	27 (69.2)	34 (87.2)	0.02
Hospitals (n=8)	5 (62.5)	8 (66.7)
Pediatric clinics (n=12)	11 (91.7)	11 (91.7)
Other miscellaneous clinics (n=19)	11 (57.9)	15 (78.9)
p-value2	0.16	0.47
5. Have the recordings of storage temperature been stored for 2 years?	22 (56.4)	24 (61.5)	0.50
Hospitals (n=8)	5 (62.5)	6 (75.0)
Pediatric clinics (n=12)	8 (66.7)	8 (66.7)
Other miscellaneous clinics (n=19)	9 (47.4)	10 (52.6)
p-value2	0.63	0.52
6. Is the refrigerator maintained at the appropriate storage temperature between 2 to 8 ?	36 (92.3)	37 (94.9)	>0.99
Hospitals (n=8)	7 (87.5)	7 (87.5)
Pediatric clinics (n=12)	12 (100)	12 (100)
Other miscellaneous clinics (n=19)	17 (89.5)	18 (94.7)
p-value2	0.58	0.46
7. Are there any varicella-containing vaccines in the refrigerator?	2 (5.1)	2 (5.1)	>0.99
Hospitals (n=8)	0 (0.0)	0 (0.0)
Pediatric clinics (n=12)	2 (16.7)	2 (16.7)
Other miscellaneous clinics (n=19)	0 (0.0)	0 (0.0)
p-value2	0.13	0.13
8. Are vaccines being stored in the center of the refrigerator?	17 (43.6)	26 (66.7)	0.004
Hospitals (n=8)	2 (25.0)	4 (50.0)
Pediatric clinics (n=12)	5 (41.7)	9 (75.0)
Other miscellaneous clinics (n=19)	10 (52.6)	13 (68.4)
p-value2	0.41	0.55
9. Are other medicines and nonmedical items (e.g., foods) being stored in the refrigerator?	14 (35.9)	8 (20.5)	0.03
Hospitals (n=8)	4 (50.0)	3 (37.5)
Pediatric clinics (n=12)	3 (25.0)	1 (8.3)
Other miscellaneous clinics (n=19)	7 (36.8)	4 (21.1)
p-value2	0.56	0.39
10. Are vaccines being separated by type and expiration date?	22 (56.4)	25 (64.1)	0.25
Hospitals (n=8)	4 (50.0)	5 (62.5)	-3
Pediatric clinics (n=12)	8 (66.7)	10 (83.3)
Other miscellaneous clinics (n=19)	10 (52.6)	10 (52.6)
p-value2	0.70	0.24
11. Have any of the vaccines passed their expiry dates?	0 (0.0)	0 (0.0)
Hospitals (n=8)	0 (0.0)	0 (0.0)	>0.99
Pediatric clinics (n=12)	0 (0.0)	0 (0.0)
Other miscellaneous clinics (n=19)	0 (0.0)	0 (0.0)
p-value2	-3	-3
12. Are multi-dose vials of vaccine being marked with the date and the time they are first opened?	29 (74.4)	27 (69.2)
Hospitals (n=8)	6 (75.0)	5 (62.5)
Pediatric clinics (n=12)	10 (83.3)	10 (83.3)
Other miscellaneous clinics (n=19)	13 (68.4)	12 (63.2)
p-value2	0.80	0.31
13. Is the refrigerator being used exclusively for vaccine storage?	03 (7.7)0	3 (7.7)	>0.99
Hospitals (n=8)	3 (37.5)	3 (37.5)
Pediatric clinics (n=12)	0 (0.0)	0 (0.0)
Other miscellaneous clinics (n=19)	0 (0.0)	0 (0.0)
p-value2	0.006	0.006
14. Is an emergency power supply available in case of power failure?	6 (15.4)	5 (12.8)	>0.99
Hospitals (n=8)	3 (37.5)	2 (25.0)
Pediatric clinics (n=12)	0 (0.0)	0 (0.0)
Other miscellaneous clinics (n=19)	3 (15.8)	3 (15.8)
p-value2	0.05	0.15
15. Is there also an extra refrigerator in case of emergency?	27 (69.2)	29 (74.4)	0.50
Hospitals (n=8)	3 (37.5)	4 (50.0)
Pediatric clinics (n=12)	11 (91.7)	11 (91.7)
Other miscellaneous clinics (n=19)	13 (68.4)	14 (73.7)
p-value2	0.03	0.13

Values are presented as no. of ‘yes’ answers (%).

¹ Changes from primary inspection (before education) in ‘yes’ answers of all 39 institutions were calculated using the McNemar χ² - test.

² Differences according to institution type were evaluated within each inspection by Fisher’s exact test of the χ² - test.

³ p-value could not be calculated because there were no ‘yes’ answers.

Table 3.

Awareness of vaccine storage and handling according to question by institution type and change after education (n=39)

Questions about knowledge of vaccine storage and handling	Before education	After education	p-value1
1. A primary vaccine administrator should be assigned	37 (94.9)	38 (97.4)	0.11
Hospitals (n=8)	8 (100)	8 (100)
Pediatric clinics (n=12)	12 (100)	12 (100)
Other miscellaneous clinics (n=19)	17 (89.5)	18 (94.7)
p-value2	0.69	>0.99
2. A refrigerator for vaccine storage only is needed	26 (66.7)	27 (69.2)	0.13
Hospitals (n=8)	7 (87.5)	7 (87.5)
Pediatric clinics (n=12)	7 (58.3)	7 (58.3)
Other miscellaneous clinics (n=19)	12 (63.2)	13 (68.4)
p-value2	0.45	0.45
3. Vaccines should be stored in a refrigerator designated only for vaccine storage	25 (64.1)	25 (64.1)	0.18
Hospitals (n=8)	6 (75.0)	6 (75.0)
Pediatric clinics (n=12)	7 (58.3)	7 (58.3)
Other miscellaneous clinics (n=19)	12 (63.2)	12 (63.2)
p-value2	0.83	0.83
4. Storing vaccines in a general refrigerator used for other purposes is all right if the temperature is maintained properly	6 (15.4)	5 (12.8)	>0.99
Hospitals (n=8)	4 (50.0)	3 (37.5)
Pediatric clinics (n=12)	00 (0.0)0	00 (0.0)0
Other miscellaneous clinics (n=19)	2 (10.5)	2 (10.5)
p-value2	0.008	0.06
5. Vaccine storage temperatures should be recorded	32 (82.0)	36 (92.3)	0.008
Hospitals (n=8)	6 (75.0)	8 (100)
Pediatric clinics (n=12)	10 (83.3)	11 (91.7)
Other miscellaneous clinics (n=19)	16 (84.2)	17 (89.5)
p-value2	0.87	>0.99
6. The refrigerator for keeping vaccines should be used only for vaccine storage	36 (92.3)	38 (97.4)	0.001
Hospitals (n=8)	7 (87.5)	8 (100)
Pediatric clinics (n=12)	12 (100)	12 (100)
Other miscellaneous clinics (n=19)	17 (89.5)	18 (94.7)
p-value2	0.58	>0.99
7. Vaccines should be kept in groups by type and expiration date	30 (76.9)	33 (84.6)	0.02
Hospitals (n=8)	6 (75.0)	6 (75.0)
Pediatric clinics (n=12)	11 (91.7)	11 (91.7)
Other miscellaneous clinics (n=19)	13 (68.4)	16 (84.2)
p-value2	0.36	0.73
8. Within a certain period after opening, multi-dose vials of vaccines should be discarded	38 (97.4)	39 (100.0)	0.11
Hospitals (n=8)	8 (100)	8 (100)
Pediatric clinics (n=12)	11 (91.7)	12 (100)
Other miscellaneous clinics (n=19)	19 (100)	19 (100)
p-value2	0.51	-3
9. Freeze/thaw cycles are allowed with frozen vaccines	36 (92.3)	38 (97.4)	0.09
Hospitals (n=8)	8 (100)	8 (100)
Pediatric clinics (n=12)	9 (75.0)	12 (100)
Other miscellaneous clinics (n=19)	19 (100)	18 (94.7)
p-value2	0.03	>0.99
10. Vaccines should not be exposed to the sun	37 (94.9)	39 (100.0)	0.03
Hospitals (n=8)	8 (100)	8 (100)
Pediatric clinics (n=12)	12 (100)	12 (100)
Other miscellaneous clinics (n=19)	17 (89.5)	19 (100)
p-value2	0.69	-3
11. The center of the refrigerator should be equipped with a thermometer	25 (64.1)	36 (92.3)	0.003
Hospitals (n=8)	6 (75.0)	8 (100)
Pediatric clinics (n=12)	7 (58.3)	11 (91.7)
Other miscellaneous clinics (n=19)	12 (63.2)	17 (89.5)
p-value2	0.83	>0.99
12. For ease of access, vaccines can be stored in the door of a refrigerator	28 (71.8)	31 (79.5)	0.002
Hospitals (n=8)	7 (87.5)	8 (100)
Pediatric clinics (n=12)	7 (58.3)	7 (58.3)
Other miscellaneous clinics (n=19)	14 (73.7)	16 (84.2)
p-value2	0.37	0.09
13. A generator is necessary in case of blackouts	23 (59.0)	27 (69.2)	0.01
Hospitals (n=8)	6 (75.0)	6 (75.0)
Pediatric clinics (n=12)	5 (41.7)	8 (66.7)
Other miscellaneous clinics (n=19)	12 (63.2)	13 (68.4)
p-value2	0.28	>0.99
14. Personnel who handle or administer vaccines need training in how to store and handle vaccines	30 (76.9)	37 (94.9)	<0.001
Hospitals (n=8)	7 (87.5)	8 (100)
Pediatric clinics (n=12)	8 (66.7)	11 (91.7)
Other miscellaneous clinics (n=19)	15 (78.9)	18 (94.7)
p-value2	0.69	>0.99

Values are presented as no. of correct answers (%).

¹ Changes from primary administration (before education) in the correct answer for all 39 participants taken together were calculated using the exact test of Wilcoxon’s signed rank test.

² Differences according to institution type were evaluated within each inspection by Fisher’s exact test of the χ² - test.

³ p-value could not be calculated because all the answers were correct.

Figure & Data

References

Citations

Citations to this article as recorded by  

• Effectiveness of the online-eLearning program KeepCoool at improving the vaccine cold chain in general practices
  Anika Thielmann, Marie-Therese Schmitz, Thomas Welchowski, Birgitta Weltermann, Pasyodun Koralage Buddhika Mahesh
  PLOS ONE.2024; 19(4): e0301847.     CrossRef
• Impfstoffmanagement in der ambulanten Versorgung: ein systematischer Review internationaler Empfehlungen und Implikationen für das deutsche Gesundheitswesen.
  Laura Knepper, Christiane Stieber, Birgitta Weltermann
  Das Gesundheitswesen.2023; 85(10): 857.     CrossRef
• Impact of educational intervention on the best immunization practices among practicing health care professionals in a south Indian city
  Juny Sebastian, Gurumurthy Parthasarathi, Mandyam Dhati Ravi
  Therapeutic Advances in Vaccines and Immunotherapy.2021; 9: 251513552110325.     CrossRef
• Improving knowledge on vaccine storage management in general practices: Learning effectiveness of an online-based program
  Anika Thielmann, Marie-Therese Puth, Birgitta Weltermann
  Vaccine.2020; 38(47): 7551.     CrossRef
• Vaccine cold chain in general practices: A prospective study in 75 refrigerators (Keep Cool study)
  Anika Thielmann, Marie-Therese Puth, Christine Kersting, Johannes Porz, Birgitta Weltermann, Sanjai Kumar
  PLOS ONE.2019; 14(11): e0224972.     CrossRef
• Visual inspection of vaccine storage conditions in general practices: A study of 75 vaccine refrigerators
  Anika Thielmann, Marie-Therese Puth, Birgitta Weltermann, Abhijit P. Pakhare
  PLOS ONE.2019; 14(12): e0225764.     CrossRef
• Safety of vaccines that have been kept outside of recommended temperatures: Reports to the Vaccine Adverse Event Reporting System (VAERS), 2008–2012
  Beth F. Hibbs, Elaine Miller, Jing Shi, Kamesha Smith, Paige Lewis, Tom T. Shimabukuro
  Vaccine.2018; 36(4): 553.     CrossRef
• Good vaccination practice: it all starts with a good vaccine storage temperature
  Frédéric Vangroenweghe
  Porcine Health Management.2017;[Epub]     CrossRef
• Effectiveness of a web-based education program to improve vaccine storage conditions in primary care (Keep Cool): study protocol for a randomized controlled trial
  Anika Thielmann, Anja Viehmann, Birgitta M. Weltermann
  Trials.2015;[Epub]     CrossRef