Drug safety is a significant concern in several countries, monitored and evaluated through pharmacovigilance systems. Pharmacovigilance is an activity related to detecting, evaluating, understanding, and preventing adverse drug reactions (ADR)1. Pharmacovigilance activities include periodic security update reports, commonly called spontaneous reporting. Spontaneous reporting of adverse events (AE)/ADR is one element of pharmacovigilance activities that help overcome safety concerns after drug administration2. This reporting activity provides actual information regarding the safety profile of real-life clinical practices compared to the results of clinical trials using only a few samples and the safety of drugs studied in a limited time3. Spontaneous reporting is a cost-effective, flexible, and highly effective method of gathering information because health workers voluntarily submit AE/ADR case reports to the National Pharmacovigilance Center of the Food and Drug, The Indonesian Food and Drug Authority (Badan Pengawas Obat dan Makanan Republik Indonesia, BPOM RI) for analysis, which will later help to reduce the potential for AE/ADR in patients4.

Spontaneous reporting is a method that contributes significantly to the improvement of pharmacovigilance in some countries5. Several regulators in other countries have designed easy systems for spontaneous reporting intending to increase the participation of health workers in spontaneous reporting, but in reality, the rate of spontaneous AE/ADR reporting is still low6,7. Spontaneous reporting of AE/ADR in several less progressive countries is of particular concern, considering that these less developed countries contribute to 80% of the disease burden in the world. However, they only participate in AE/ADR reports in less than 1% of all global reports (11,824,804)8.

Little participation in AE/ADR reporting in less developed countries such as Indonesia cannot be explained because the law does not require it. Spontaneous reporting of AE/ADR, reporting still needs to be improved9. The low level of AE/ADR reporting is caused by several factors, such as not caring about patient safety, feeling they have no responsibility to report, complicated reporting procedures, and poor knowledge and attitudes of health workers, which impacts AE/ADR reporting practices10. However, research on factors that correlate with spontaneous reporting practices in Indonesia still needs to be improved.

Therefore, research related to this needs to be done. Because research related to the factors that influence the practice of spontaneous reporting is still new, it requires instruments with good validity and reliability so that the data collection process is accurate and precise. Research instruments for observational research that look at the factors that influence a person generally use questionnaires because questionnaires are relatively easy to collect data in research and policy evaluation. Information containing knowledge, attitudes, opinions, behavior, and facts will be easily collected using a questionnaire11. Of course, a questionnaire with good validity and reliability requires a development process first from evaluating the validity and reliability of test results12.

Validity and reliability are the two most essential and fundamental features in evaluating any measurement instrument or tool for good research, one of which is a questionnaire. Without assessing the reliability and validity of the study, it will be difficult to describe the effects of measurement errors on the theoretical relationships being measured13. Research with a valid questionnaire will produce data that follows the construct built by the researcher. As for reliability, it serves to minimize measurement errors from questionnaires when taking data. Reliability is an indicator of questionnaire consistency when measuring certain concepts12.

Given the importance of validity and reliability testing for a research questionnaire, it should be a priority before taking data. However, a review of articles from 748 studies found that one-third did not attach procedures to establish validity (31%) or reliability (33%). Meanwhile, developing accurate and precise questionnaires is needed to decrease measurement errors, namely mismatches between respondent attributes and survey responses14. Validity and reliability tests must be carried out before data collection to reduce measurement errors and measure the questionnaire's reproducibility, and the results will be evaluated later. Therefore, this study aims to test the validity and reliability of a questionnaire developed to measure pharmacists' knowledge and management of spontaneous reporting.

Materials

The instrument in this study was a questionnaire created by researchers based on World Health Organization (WHO) and BPOM RI pharmacovigilance guidelines15-20. In addition, the results of expert consultation were considered. The questionnaire consisted of five parts: pharmacist demographic characteristics, knowledge included in individual variables, psychological variables, environmental variables, and the practices that pharmacists engaged in spontaneous reporting. For the demographics section, there were 15 question items. For the part of knowledge included in individual variables, there were 13 questions. For psychological variables, there were 14 questions. Environmental variables totaled nine questions, and practice summed 19 questions. The total number of questions in the questionnaire was 55 questions (Table I). The questionnaire was validated in two stages: the first stage was face validation for the overall appearance of the questionnaire, and the second stage was construct validation to see whether the questionnaire could produce data following the construct developed by the researcher. Constructing validation in this questionnaire was divided into two methods. The first method was biserial point correlation for the knowledge section because the answer scale was the Guttman scale or dichotomy21. Because the answer scale used an ordinal scale, the factor analysis method was applied for psychological, environmental, and practice factors22.

TableI. Question items in each instrument domain.

Methods

Design and participant

This study was conducted between September and October 2023 in pharmacists working full-time at type C hospitals in Surabaya and Sidoarjo, East Java, Indonesia. Type C hospitals were chosen because the number of type C hospitals is the largest in Indonesia, but the contribution of reports of adverse drug reactions is low16,23. There were 30 samples recommended by Hertzog24. The Health Research Ethics Commission (KEPK), Faculty of Dentistry, Universitas Airlangga, issued the certificate of ethical eligibility for this study, with number 987/HRECC. FODM/VIII/2023.

Face validity

The method used to assess face validity was to provide a suggestion and improvement column at the end of the questionnaire to comment on all parts of the questionnaire in terms of language, font size, font, and word choice. Face validity was a subjective assessment of the operation of a construct. A test was valid if its content seemed relevant to the person working on it. It evaluated the appearance of the questionnaire in terms of feasibility, readability, consistency of style and format, and clarity of the language used. In other words, face validity referred to the researcher's subjective assessment of the presentation and relevance of the measuring instrument, whether the items appeared relevant, reasonable, clear, and transparent25.

Data analysis

Construct validity test

The construct validity test used two methods to determine the questionnaire's construct conformity. The first method used the biserial point correlation for the knowledge section, and the second used factor analysis for psychological, environmental, and practice variables questionnaires. For biserial point correlation to check the validity of the knowledge section, the method using Microsoft Excel tools to show the difference between the r-value and the r-table, in which Rpb was biserial point correlation coefficient, xi was average total score of respondents who answered correctly, xt was average total score of all respondents, Pi was proportion of correct answers item i, Qi was 1-Pi, and St was standard deviation of the total score, as shown in Equation 1.Equation 1 was used to calculate the r-value to show the validity of the questionnaire item26. This calculation was applied to each knowledge question item. The question item was considered valid if the computed r-value offered a value greater than the r-table, and vice versa.

[1]

The second method was factor analysis using several indicators using IBM SPSS Statistics 26 (https://www.ibm.com/support/pages/downloading-ibm-spss-statistics-26). The first indicator was the Kaiser-Meyer-Olkin (KMO) measurement. This parameter compared the correlation coefficient value with the partial coefficient value. The requirement for factor analysis was that the KMO value had to be higher than 0.5. Bartlett's Test of sphericity tested the dependence between the variables being tested. This parameter helped indicate the absence of correlation between variables with each other in the community. The significance value in Bartlett's Test had to be less than 0.05 so that the process could continue for factor analysis. The following indicator showed the result of the calculation of the anti-image correlation. Grades with 'A' indicated the Measure of Sampling Adequacy (MSA) value. If the MSA number for a variable was below 0.5, then the variable had to be excluded, and variable selection had to be repeated. The last indicators were the Component Matrix and Rotated Component Matrix, which helped explain the spread of variables into factors formed. The Component Matrix confirmed whether or not there was a correlation between items and components. A high correlation value showed a solid relationship between the items and the components so that the items could be used as a factor. In a complex matrix, interpreting these factors was quite rare because it was difficult. Therefore, the factor alteration used in matrix factor rotation was converted into a more friendly form to understand. The steps for factor analysis in SPSS were selecting the analysis menu, selecting dimension and factor reduction, selecting the variables to be analyzed, selecting the descriptive option, checking the initial solution, KMO, Bartlett's Test of sphericity, and anti-image, selecting OK, then selecting the rotation menu to select varimax and check rotated solution and loading plots, and finally selecting OK for the analysis process27.

Reliability test

Reliability testing aimed to see how consistent a questionnaire was when used for data collection. The method used was to look at the value of Cronbach α, which had to be greater than 0.6 for qualified reliability28. However, another theory was that if the Cronbach α value was 0.5-0.7, the questionnaire could be considered moderately reliable and still be used for research data collection29,30. The way to carry out reliability analysis in IBM SPSS Statistics 26 tools was to select the analyze menu, click scale and select reliability analysis, then choose the variable to be measured, click the statistics menu, and check the scale if the item deleted option, then clicked ok to process the analysis, later the results would display the Cronbach α value of analyzed variables.

This questionnaire pilot test found 30 pharmacist respondents who worked in type C hospitals in Surabaya and Sidoarjo. The majority of respondents were 28 (93.3%) female. Most respondents were 28 (93.3%) from type C hospitals in Surabaya and 2 (6.6%) from type C hospitals in Sidoarjo. Face validity shows that overall, the pilot test respondents said that the grammar of the questionnaire was excellent and easy to understand, and the sentence structure was not ambiguous so that respondents could understand the meaning of the questions on the questionnaire. The font size and typeface used are also ideal, according to respondents. The form validation method is similar to testing the validity of questionnaires conducted in India when testing the validity of work-related stress questionnaires (TAWS-16)31.

The results of construct validation of the knowledge question item section that is included in individual variables can be seen in Table II. The calculation results in Table II are interpreted by comparing the calculated r-value with the r-table. The question item is arguably valid if the calculated r-value exceeds the r-table32. Based on the data above, two knowledge question items show invalid results because the calculated r-value is smaller than the r-table: question items number 4 and 13. Several factors can cause the invalidity of the question item. The first possibility is that the question item contains a sentence that leads the respondent to lean toward one answer choice, or the other option is that the question item cannot describe the intention the researcher wants to ask. Hence, the respondent gives an inappropriate response33.

TableII. Knowledge question item validation results.

* invalid item

The next validity test is for psychological, environmental, and practical variables, using factor analysis methods. The first indicator shows the value of the KMO Measure and the significance values of psychological, environmental, and practice variables. The KMO and Bartlett's tests are data suitability tests that must be performed before interpreting the factor analysis results. The MSA is a statistical value that indicates the proportion of diversity in the variables on which factor analysis is based34. If the MSA value >0.50, It is concluded that the questionnaire can be used to measure respondents' answers precisely. If it shows a KMO value of more than 0.5 and a significance value of less than 0.05, the variable can be used for further data collection and analysis27.

Bartlett's test examines whether the indicators used correlate and are suitable for factor analysis. If the value of Bartlett's test is less than 0.05, it is concluded that the indicators used are correlated and ideal for factor analysis. The KMO values of psychological, environmental, and practice variables are 0.535 each, 0,582, and 0.634, with each significant value below 0.05. These results show that the indicators used in this study are correlated and appropriate for factor analysis27.

Factor analysis requires the data matrix to correlate factor analysis. The correlation value is shown in the anti-image correlation matrix. The MSA value on the diagonal anti-image correlation with the sign is expected to be above 0.527. Table IIIshows that each variable has a question item whose value is less than 0.5. The first of the psychological variables shows that items 4, 8, and 13 have values less than 0.5, which are 0.229, 0.386, and 0.306. The second environmental variable is shown in item number 3; whose value is less than 0.5, which is 0.435. Finally, from the practice variables, there are four items whose value is less than 0.5: items 1, 13, 18, and 19. Based on these results, question items with an MSA value of less than 0.5 cannot be continued for the data retrieval process, while other items with an MSA value of more than 0.5 can be used for the data retrieval process22.

The results of the subsequent construct validation can be seen in Table IV. Table IV presents loading factor coefficient data explaining the connection between the origin variable and the factor. A significant correlation value denotes a solid relationship between the factor and the original variable, which means that the variable can be used as a factor. In a complex matrix, interpreting these factors is quite rare because it is difficult. Therefore, the factor alteration used in matrix factor rotation is converted into a more friendly form to understand36.

Rotated Component Matrix is the value of the distribution of variables that have been extracted into factors that are formed based on the loading factor after the transformation process to a form that is easier to understand. The loading factor value could turn after the process rotation. Component variables with a loading factor of less than 0.5 are deemed not to contribute to the factors formed significantly, so they must be eliminated from the factors formed27. However, for this loading factor value, there is a theory that says if the value is more than 0.3, then the item has shown a close relationship between items on the factor formed35. However, in this study, all items on each variable have a loading factor value of more than 0.5, meaning that all indicator items have a close relationship with the factors formed36.

TableIII. Measurement results of MSA of psychological, environment, and practice variables.

* item does not meet the requirement

TableIV. Results of loading factor measurement of psychological, environmental, and practice variables from the Rotated Component Matrix.

* item does not meet the requirement

The calculated component transformation analysis results must support the results in Table IV. Suppose the component value of a variable shows a value that is large or more than 0.5. In that case, the relationship between the factors or components that make up a variable is getting closer37. Based on the component transformation matrix calculation, psychological variables are divided into four components. Still, component number three has a value of less than 0.5, so component number three is considered not to describe the construct of psychological variables. In Table IV, the psychological variables section results from factor loadings on psychological variables, which are included in factor or component 3 in items 8, 12, and 13, which are not included as components that make up psychological variables. Likewise, for environmental variables where the results of the transformation component matrix are only components 1 and 2, which have a solid correlation, meaning that the environmental variable question items included in components 3 and 4 are considered weak variable constituents, therefore environmental variable question items number 3, 6, and 8 are deemed unable to represent environmental variables. In the last part, based on the results of the component transformation matrix, the training variables show that only components 1 and 5 have a strong correlation, meaning that the training questions included in components 2, 3, 4, and 6 cannot represent the training variables.

The reliability test aims to see how consistent a questionnaire item is when tested on several research samples. The reliability value is potentially high if each item has a close correlation38. The Cronbach α value must be greater than 0.6 to be eligible for reliability28. Nevertheless, another theory39 says that if the value of Cronbach α is 0.4 to 0.6, it can be reliable, calculated, and used for data collection. The reliability test results in Table V show that the Cronbach α value of all variables is classified as reliable because the value is more than 0.6, meaning that question items from psychological, environmental, and practice variables are reproducible and worthy of being used as research instruments28. However, the corrected item's total correlation value is another parameter to see a question item's reliability.

TableV. Cronbach α value of psychological, environmental, and practice variables.

The function of the corrected item-total correlation value is to select items whose measuring function is under the test measuring function as the compiler desires. In other words, it is to choose an item that measures the same thing as what the test as a whole measure40. According to Azwar41, a coefficient limit of >0.30 is commonly employed as a criterion for selecting items based on item-total correlation. As part of the test, all items with a correlation coefficient of at least 0.30 were certified psychometrically eligible. However, another theory42 says that the item-total correlation value must be greater than the r-table to be reliable. After analysis, results show unqualified values based on the two theories above; in psychological variables, item 4 shows values less than 0.3 and more minor than the r-table. For environment variables, items 2, 3, 5, and 9 indicate low values, then practice items 6, 18, and 19, whose values do not qualify. Therefore, some of these items can be removed from the questionnaire.

The results of this validity and reliability test aim to select question items suitable for use in the data collection process because they relate to the purpose of the questionnaire, which is to get answers under the construct built into the questionnaire. Regarding the validity and reliability test results, there is a theory that states that every valid questionnaire question item must also have good reliability because if the item is accurate, then the reproducibility is also good. Unlike reliability, not all questionnaires with good reliability will result in valid question items because the accuracy of the answers has not been tested43.

Based on the validity and reliability test results, several question items must be eliminated because they cannot provide accurate answers according to the variable construct created. The final form of the questionnaire, which has been evaluated for validity and reliability, can be seen in Table VI. From the 55 initial question items, the number was reduced to only 34, and this was because 21 question items in the questionnaire did not meet the validity and reliability requirements.

This research has limitations, and the sample size is only 30 respondents because few pharmacists in type C hospitals are willing to be pilot test respondents. This small number of respondents causes a lack of representation, potentially affecting the results' validity and reliability44. However, on the other hand, this research has strength. The strength of this research is that there are no open questions in the questionnaire developed, so the respondents' answers are common to process. Apart from that, the analysis used to test the construct validity of the questionnaire is relatively common because most questions use an ordinal answer scale, so the researcher can use the factor analysis method to construct validity.

Regardless of the strengths and limitations above, this questionnaire benefits researchers in finding out the factors that influence spontaneous reporting practices by pharmacists because this questionnaire can provide accurate, precise, and reproducible results. This questionnaire can be used for pharmacist respondents who work in type C hospitals in East Java. Suppose this questionnaire will be used for pharmacists in other types of hospitals or health services, such as community health centers or drug stores, or for pharmacists outside East Java. This questionnaire can be used but requires a verification process to adapt it to the pharmacist's workplace and location. Suggestions for the next step when developing a research questionnaire instrument: the researcher must start with making the correct conceptual framework design, compiling the questions that the researcher wants to make in the questionnaire, and determining what type of question-answer it looks like, then make a filter that suits the target respondent, then eliminate various potential biases and double questions in one question item. Then, it also made a picture of what the analysis will be like and the last and main one that pilots must test before being used for research45.

TableVI. Question items in each instrument domain after evaluation of validity and reliability results.

* invalid item

Evaluation of validation results on knowledge question items found two invalid items and four question items could not represent psychological variables. There are three invalid items for environmental variables, and then for practice variables, there are twelve invalid items. Thirty-four question items can still be used to acquire further data.

The author would like to thank the directors of type C hospitals in Surabaya and Sidoarjo for allowing questionnaires to be distributed within their facilities. Furthermore, all pharmacists in type C hospitals were eager to fill out questionnaires and participate in this study. The author also expresses gratitude to Universitas Airlangga's Center of Excellence for Patient Safety and Quality, which assisted in funding the research. Also, thanks to the Universitas Airlangga's Innovative Pharmacy Practice and Integrated Outcome Research (INACORE) Group for providing recommendations and advice on the manuscript's writing and content.

Conceptualization: Favian Rafif Firdaus, Elida Zairina

Data curation: Favian Rafif Firdaus

Formal analysis: Favian Rafif Firdaus, Elida Zairina

Funding acquisition: -

Investigation: Favian Rafif Firdaus

Methodology: Elida Zairina

Project administration: Favian Rafif Firdaus

Resources: Elida Zairina

Software: -

Supervision: Yunita Nita, Catur Dian Setiawan, Elida Zairina

Validation: Elida Zairina

Visualization: -

Writing - original draft: Favian Rafif Firdaus

Writing - review & editing: Yunita Nita, Catur Dian Setiawan, Elida Zairina

None.

There are no conflicts of interest.

1. Jose J, Al Rubaie MH, Al Ramimmy H, Varughese SS. Pharmacovigilance: Basic concepts and an overview of the system in Oman. Sultan Qaboos Univ Med J. 2021;21(2):e161-3. DOI: 10.18295/squmj.2021.21.02.001; PMCID: PMC8219341; PMID: 34221460

2. Zhou N, Fang Z, Liang D, Lian J, Han H, Li R. Hospital information system-based pharmacovigilance for elderly patients: practices and results. J Pak Med Assoc. 2023;73(3):525–32. DOI: 10.47391/jpma.6428; PMID: 36932754

3. Katyal J, Arora E, Gupta YK. Impact of increased focus on pharmacovigilance on knowledge and attitude towards adverse drug reaction reporting among medical students in India. Int J Risk Saf Med. 2020;31(1):15–24. DOI: 10.3233/jrs-195012; PMID: 31594254

4. Asiamah M, Akuffo KO, Nortey P, Donkor N, Danso-Appiah A. Spontaneous reporting of adverse drug reaction among health professionals in Ghana. Arch Public Health. 2022;80(1):33. DOI: 10.1186/s13690-021-00783-1; PMCID: PMC8772084; PMID: 35057859

5. Yawson AA, Abekah-Nkrumah G, Okai GA, Ofori CG. Awareness, knowledge, and attitude toward adverse drug reaction (ADR) reporting among healthcare professionals in Ghana. Ther Adv Drug Saf. 2022;13:20420986221116468. DOI: 10.1177/20420986221116468; PMCID: PMC9364224; PMID: 35966898

6. Bahlol M, Bushell M, Khojah HMJ, Dewey RS. Spontaneous adverse drug reaction reporting by community pharmacists: preparedness and barriers. Saudi Pharm J. 2022;30(7):1052–9. DOI: 10.1016/j.jsps.2022.04.006; PMCID: PMC9315256; PMID: 35903525

7. Kobayashi T, Noda A, Obara T, Tsuchiya M, Akasaka K, Yoshida M, et al. Knowledge, Attitudes, and Practice of Hospital Pharmacists Regarding Pharmacovigilance and Adverse Drug Reaction Reporting in Japan. Hosp Pharm. 2021;56(1):7–16. DOI: 10.1177/0018578719851455

8. Ampadu HH, Hoekman J, de Bruin ML, Pal SN, Olsson S, Sartori D, et al. Adverse Drug Reaction Reporting in Africa and a Comparison of Individual Case Safety Report Characteristics Between Africa and the Rest of the World: Analyses of Spontaneous Reports in VigiBase®. Drug Saf. 2016;39(4):335–45. DOI: 10.1007/s40264-015-0387-4; PMCID: PMC4796322; PMID: 26754924

9. Kopciuch D, Zaprutko T, Paczkowska A, Ratajczak P, Zielińska-Tomczak Ł, Kus K, et al. Safety of medicines—Pharmacists' knowledge, practice, and attitudes toward pharmacovigilance and adverse drug reactions reporting process. Pharmacoepidemiol Drug Saf. 2019;28(12):1543–51. DOI: 10.1002/pds.4792; PMID: 31148344

10. Hailu AD, Mohammed SA. Adverse Drug Reaction Reporting in Ethiopia: Systematic Review. Biomed Res Int. 2020;2020:8569314. DOI: 10.1155/2020/8569314; PMCID: PMC7439161; PMID: 32851089

11. Bhattacharyya S, Kaur R, Kaur S, Ali SA. Validity and reliability of a questionnaire: a literature review. Chron Dent Res. 2017;6(2):16-22.

12. Mohajan HK. Two Criteria for Good Measurements in Research: Validity and Reliability. Ann Spiru Haret Univ Econom Ser. 2017;17(4):59-82. DOI: 10.26458/1746

13. Ramu P, Osman M, Mutalib NAA, Aljaberi MA, Lee KH, Lin CY, et al. Validity and Reliability of a Questionnaire on the Knowledge, Attitudes, Perceptions and Practices toward Food Poisoning among Malaysian Secondary School Students: A Pilot Study. Healthcare. 2023;11(6):853. DOI: 10.3390/healthcare11060853; PMCID: PMC10047946; PMID: 36981510

14. Radhakrishna RB. Tips for Developing and Testing Questionnaires/Instruments. J Ext. 2007;45(1):25.

15. World Health Organization. The importance of pharmacovigilance. Geneva: World Health Organization; 2002. p. 48.

16. Badan Pengawas Obat dan Makanan Republik Indonesia. Buletin Berita MESO Juni 2023 Vol 41 No. 1. Jakarta: Badan Pengawas Obat dan Makanan Republik Indonesia; 2023.

17. Badan Pengawas Obat dan Makanan Republik Indonesia. Peraturan Badan Pengawas Obat dan Makanan Nomor 15 Tahun 2022 tentang Penerapan Farmakovigilans. Jakarta: Badan Pengawas Obat dan Makanan Republik Indonesia; 2022.

18. Badan Pengawas Obat dan Makanan Republik Indonesia. Modul Farmakovigilans Dasar: Project for Ensuring Drug and Food Safety. Jakarta: Badan Pengawas Obat dan Makanan Republik Indonesia; 2020. Available from: https://e-meso.pom.go.id/web/useruploads/files/reference/5eaa2698c3abc_Modul%20Farmakovigilans%20Dasar.pdf

19. Badan Pengawas Obat dan Makanan Republik Indonesia. Modul Farmakovigilans untuk Tenaga Profesional Kesehatan: Project for Ensuring Drug and Food Safety. Jakarta: Badan Pengawas Obat dan Makanan Republik Indonesia; 2020. Available from: https://e-meso.pom.go.id/web/useruploads/files/reference/5efb0dbc8d7eb_MODUL%20FARMAKOVIGILANS%20BAGI%20TENAGA%20PROFESIONAL%20KESEHATAN.pdf

20. Badan Pengawas Obat dan Makanan Republik Indonesia. Lampiran – Daftar Produk yang Mengandung Policresulen Cairan Obat Luar Konsentrat. Jakarta: Badan Pengawas Obat dan Makanan Republik Indonesia; 2018. Available from: https://www.pom.go.id/storage/dat/20180215/Lampiran_penjelasan_Kabadan.pdf

21. Guest G. Using Guttman Scaling to Rank Wealth: Integrating Quantitative and Qualitative Data. Field Methods. 2000;12(4):346-57. DOI: 10.1177/1525822X0001200406

22. Setiawan MH, Supriyadi, Bharati DAL. Analysis of Instrument Development Recount Text Writing Test HOTS for Vocational High School using Exploratory Factor Analysis (EFA). J Educ Sci Technol. 2022;8(1):74-86. DOI: 10.26858/est.v8i2.25910

23. Trisnantoro L, Listyani E. Jumlah Rumah Sakit di Indonesia Pertumbuhan Rumah Sakit Publik. Jakarta: Perhimpunan Rumah Sakit Seluruh Indonesia; 2018. Available from: https://web.persi.or.id/analis-data/441-jumlah-rs-di-indonesia-pertumbuhan-rs-publik

24. Hertzog MA. Considerations in determining sample size for pilot studies. Res Nurs Health. 2008;31(2):180–91. DOI: 10.1002/nur.20247; PMID: 18183564

25. Oluwatayo JA. Validity and Reliability Issues in Educational Research. J Educ Soc Res. 2012;2(2):391-400.

26. Brown JD. Statistics corner: Questions and answers about language testing statistics: Point-biserial correlation coefficients. Shiken. 2001;4(3):13-7.

27. Santoso S. Analisis SPSS pada statistik parametrik. Jakarta: Elex Media Komputindo; 2012.

28. Streiner DL. Starting at the beginning: An introduction to coefficient alpha and internal consistency. J Pers Assess. 2003;80(1):99–103. DOI: 10.1207/s15327752jpa8001_18; PMID: 12584072

29. Basuki I, Hariyanto. Asesemen pembelajaran. Bandung: Remaja Rosdakarya; 2014.

30. Surapranata S. Analisis, Validitas, Reliabilitas dan Interpretasi Hasil Tes Implementasi Kurikulum 2004. Bandung: Remaja Rosdakarya; 2009.

31. Roy R, Sukumar GM, Philip M, Gopalakrishna G. Face, content, criterion and construct validity assessment of a newly developed tool to assess and classify work–related stress (TAWS– 16). PLoS One. 2023;18(1):e0280189. DOI: 10.1371/journal.pone.0280189; PMCID: PMC9821462; PMID: 36608043

32. Perinetti G. StaTips Part VI: Bivariate correlation. South Eur J Orthod Dentofac Res. 2019;6(1):2–5. DOI: 10.5937/sejodr6-21664

33. Bonett DG. Point-biserial correlation: Interval estimation, hypothesis testing, meta-analysis, and sample size determination. Br J Math Stat Psychol. 2020;73(Suppl 1):113–44. DOI: 10.1111/bmsp.12189; PMID: 31565811

34. Chan LL, Idris N. Validity and Reliability of The Instrument Using Exploratory Factor Analysis and Cronbach’s alpha. Int J Acad Res Bus Soc Sci. 2017;7(10):400-10. DOI: 10.6007/IJARBSS/v7-i10/3387

35. Purwanto. Instrumen penelitian sosial dan pendidikan : pengembangan dan pemanfaatan. Yogyakarta: Pustaka Pelajar; 2007.

36. Bewick V, Cheek L, Ball J. Statistics review 7: Correlation and regression. Crit Care. 2003;7(6):451-9. DOI: 10.1186/cc2401; PMCID: PMC374386; PMID: 14624685

37. Wu RMX, Zhang Z, Zhang H, Wang Y, Shafiabady N, Wan W, et al. An FSV analysis approach to verify the robustness of the triple-correlation analysis theoretical framework. Sci Rep. 2023;13(1):9621. DOI: 10.1038/s41598-023-35900-3; PMCID: PMC10267157; PMID: 37316559

38. Sürücü L, Maslakçi A. Validity and Reliability in Quantitative Research. Bus Manag Stud Int J. 2020;8(3):2694–726. DOI: 10.15295/bmij.v8i3.1540

39. Rusilowati A. Pengembangan Instrumen Penilaian. Semarang: UNNES Press; 2014.

40. Hendryadi. Editorial Note: Uji Validitas Dengan Korelasi Item-Total? J Manajemen Strategi Aplikasi Bisnis. 2021;4(1):315–20. DOI: 10.36407/jmsab.v4i2.404

41. Azwar S. Reliabilitas dan Validitas. 4th ed. Yogyakarta: Pustaka Pelajar; 2012.

42. Slamet R, Wahyuningsih S. Validitas dan Reliabilitas terhadap Instrumen Kepuasan Kerja. J Manajemen Bisnis Aliansi. 2022;17(2):51–8. DOI: 10.46975/aliansi.v17i2.428

43. Bolarinwa OA. Principles and Methods of Validity and Reliability Testing of Questionnaires Used in Social and Health Science Researches. Niger Postgrad Med J. 2015;22(4):195-201. DOI: 10.4103/1117-1936.173959; PMID: 26776330

44. Lakens D. Sample Size Justification. Collabra Psychology. 2022;8(1):33267. DOI: 10.1525/collabra.33267

45. Jenn C. Designing a Questionnaire. Malays Fam Physician. 2006;1(1):32-5.