This is an open access article under the CC-BY-SA license. REiD (Research and Evaluation in Education), 5(2), 2019, 144-151 Available online at: http://journal.uny.ac.id/index.php/reid The respondent factors on the digital questionnaire responses *1Muhardis; 2Burhanuddin Tola; 3Herwindo Hariwibowo 1,2,3Department of Educational Research and Evaluation, Universitas Negeri Jakarta Jl. R. Mangun Muka, Rawamangun, Pulo Gadung, Jakarta Timur, DKI Jakarta 13220, Indonesia *Corresponding Author. E-mail: adi_perdana2000@yahoo.com Submitted: 2 September 2019 | Revised: 19 November 2019 | Accepted: 27 November 2019 Abstract Progress in the field of technology often facilitates human work. One of them is progress in the development of questionnaire modes. Currently, existing questionnaires have been based on a digital platform, which makes evaluators easy to design, disseminate, and conduct scoring. All are computer- based, making them reachable by the respondents no matter how far the location of the respondent is, as long as they are connected to the internet. However, any progress is accompanied by several obstacles. For example, the respondents experienced an error in responding to having the intent to respond 'Yes' option but pressing the 'No' button instead. It is very different from filling in paper and pencil based questionnaires in which they are sure to put a checkmark using a pencil on the answer choices. This problem is what the researchers found when distributing digital questionnaires to participants of the National Questions Writing Program based on the 'SIAP' (Sistem Inovatif Aplikasi Penilaian) application. On conditional questions (if you choose 'No', please stop), some respondents who have chosen 'No' answers still respond to the next questions. It causes the data obtained are unreliable. After conducting a more in-depth analysis, the researchers found that respondents’ factors as psychological factors are the cause, such as the new experience of accessing applications, understanding of applications, stress, and personal health. Uniquely, the respondents who have problems are those in the context of productive age, i.e 30 to 39 years old, more than five years of teaching experience, postgraduate level, and female. Keywords: computer-based assessment, digital questionnaire, respondent factors, SIAP application Permalink/DOI: https://doi.org/10.21831/reid.v5i2.26943 Introduction Progress in the field of assessment is increasingly rapid. Assessment is no longer carried out in conventional ways, such as paper-based test (PBT) but has been in the form of a computer-based test (CBT). Many experts argue that well-designed computer- based assessments are considered more com- prehensive, more accurate, and able to de- scribe the profile of each test participant (Nezami & Butcher, 2000). Computer-based assessment is also the first step in the test model for the future (Yanxia, 2017) by not ignoring other aspects of the application that can cause the computer to make an error in processing the response given (McArthur & Choppin, 1984). Errors are predictable and deserve more attention than the application of CBT, not only errors from the application side but also those from the respondent's side. Web-based questionnaires, as one of the other forms of application of CBT, as they have been used in Australia (Nulty, 2008), also need to pay at- tention to these errors. Before being applied, it is better for prospective respondents to be given socialization, training, and how to use the application (Sevillano-García & Vázquez- Cano, 2015). Web-based questionnaires, also known as digital questionnaires, need to pay attention to some aspects such as questions asked and visual design so respondents are more aware in answering so that there are no mistakes https://doi.org/10.21831/reid.v5i2.26943 The respondent factors on the digital questionnaire... Muhardis, Burhanuddin Tola, & Herwindo Hariwibowo Copyright © 2019, REiD (Research and Evaluation in Education), 5(2), 2019 - 145 ISSN 2460-6995 when answering. Excessive use of media, such as quality-check reminders, will take a long time which can make the respondents frustra- ted and leave the questionnaire prematurely (Reja, Manfreda, Hlebec, & Vehovar, 2003). It also happened to the web-based questionnaire used by the Center for Educa- tional Assessment or Pusat Penilaian Pendidikan (Puspendik) as an instrument for measuring the level of satisfaction with the implementation of the national question writing application based on SIAP. As a new program that has only been operating since 2017, SIAP needs to be evaluated to see its effectiveness. Evaluation of the participants’ satisfac- tion level was done using a web integrated with the SIAP application. On the question- naire’s initial page, it was stated that the ques- tionnaire was arranged to increase the capabil- ity of SIAP in providing information tech- nology services to users, especially writers, reviewers, and material administrators. There are two types of responses provided in this questionnaire: Yes/No answers and gradation of five Likert scales approval levels (SIAP Ap- plication Development Questionnaire, n.d.). It is interesting to find based on the response that there were respondents who still answered follow-up questions that they should not answer if they chose the response 'No'. An example can be seen in Table 1. Table 1. Communication results with the person in charge of the subject matter Choice of Answer Number of Respondents (n=295) Percentage (%) Yes No Abstain 200 92 3 47.96 21.82 30.21 Table 1 is the result of question data regarding communication with the person in charge (PIC) of the subject matter. Of the 200 respondents, 92 people answered that they did not communicate with the PIC of the subject matter. However, of the 92 people, 30 of them (32.60%) still answered the follow-up questions, namely regarding clarity of indi- cators, improvements, and rejections. It shows that web-based questionnaires require the ability of 'extensive reading' rather than 'speed reading'. In fact, more than 60% of respondents are postgraduate students. Thus, the habit of delaying the task of reading during education (Onwuegbuzie et al., 2004) or the habit of skimming reading (Cheng & Tsai, 2017) can be the cause. The respondents who graduated from the Bahasa Indonesia major did not show a much different pattern. In addition, most of them are male. Indeed, the results of the study by Parquette (1952) show that men find it difficult to understand a text in the first chance of reading. Data that are duly not filled, but still get the response, as mentioned earlier, is said to be data outliers or outlier data. These data cause the shrinking parameter to be zero so the model used becomes inappropriate, or there is a bias in the parameters (Martel, 2015). Further, these outliers can also influ- ence subsequent observations which have im- plications for the resulting parameter estima- tors (McQuarrie & Tsai, 2003). It may cause fatal errors if the next statistical analysis cannot be performed optimally only because of the data outliers. This study focuses on the context 'at- tached' to the outliers: explaining the context behind the outliers’ emergence. The context, in this case, refers to the things attached to the respondent, such as gender, age, educa- tion level, and length of teaching time. In contrast to studies conducted by psychomet- rists, such studies on outliers only focused on the outliers location in the data (Ali, 1994), statistical behavior tests (Prescott, 1978), and outliers’ patterns in the contingency tables (Rapallo, 2012), this study looks at outliers from a qualitative perspective, which uses statistical data, that are, nonparametric scale. Method Data on the outliers and respondents' identity (context) were obtained from second- ary data (Harris, 2001) from the web ques- tionnaire (SIAP Application Development Questionnaire, n.d.) in the SIAP application. The respondents are 295 people from nation- al question writers from national selection re- sults in 2016 and 2017. The number of re- spondents sampled was all respondents who answered 'No', but still answered follow-up The respondent factors on the digital questionnaire... Muhardis, Burhanuddin Tola, & Herwindo Hariwibowo 146 - Copyright © 2019, REiD (Research and Evaluation in Education), 5(2), 2019 ISSN 2460-6995 questions for three questions, 106 respon- dents for questions took a long time accessing the SIAP application, 92 respondents for communication questions with the subject matter PIC, and four people for communica- tion questions with the SIAP administrator. The questionnaire used as the data source is in the form of a digital questionnaire consisting of five parts, namely, the first part containing the identity of the respondents (name, subject, post, city, province, age, gen- der, the experience of accessing SIAP appli- cations, length of teaching, level of education, suitability for subjects taught), the second part in the form of questions about the time need- ed to access the SIAP application, the third part in the form of questions about communi- cation with the PIC of the subject matter, the fourth part which is a question regarding communication with the SIAP administrator, and the last part in the form of suggestion. The stages of research are guided by evalu- ative methods (Ross & Cronbach, 1976) and (Fíncher, 1981), namely by reading-intensive results of the response, especially responses related to questions that have further ques- tions; recording into an identity data card (in this case the context in the form of gender, recent education, and length of teaching time) of respondents who are outliers; doing coding based on context; grouping the data; giving meaning to the context of outliers; drawing a conclusion; and compiling reports. Findings and Discussion The analysis was done on question- naires distributed to national question writers about the satisfaction level with the SIAP ap- plication, communication with the subject matter PIC, and communication with the ad- ministrator. The questions on the SIAP appli- cation include the need to access the SIAP ap- plication for a long time, interactive features, and overall satisfaction with the SIAP applica- tion. The level of satisfaction with regard to communication with the PIC of the subject matter contains questions about the explana- tion of the indicators, remedial comments, and explanations of rejection. Communication with the administrator contains questions re- lated to the call center, WhatsApp, and email. Table 2. Response to the SIAP access time, communication results with the PIC of the subject matter and the SIAP administrator SIAP Access Time (n=295) Communication with the PIC of the Subject Matter (n=295) Communication with SIAP Admin (n= 295) Number of respondent answered Yes 189 200 266 Number of respondent answered No 106 92 4 Number of abstain respondent answered follow-up questions 86 30 3 a. Gender Male 44 13 1 Female 42 17 2 b. Level of education Graduate 46 17 - Post graduate 40 13 3 c. Age 25 to 29 years old 6 - - 30 to 39 years old 37 14 1 40 to 49 years old 36 11 1 50 years old or more 6 4 1 d. Length of teaching time 1 year 19 6 1 2 years 10 3 - 3 years 9 4 - 4 years 2 2 - 5 years or more 45 15 2 The respondent factors on the digital questionnaire... Muhardis, Burhanuddin Tola, & Herwindo Hariwibowo Copyright © 2019, REiD (Research and Evaluation in Education), 5(2), 2019 - 147 ISSN 2460-6995 Based on the results of the analysis, in- formation was obtained on the percentage of respondents who answered 'No' but still an- swered the follow-up questions for the three types of questions, respectively 81.13% (for questions that took a long time accessing the SIAP application), 32.6% (for communication questions with the PIC of the subject matter, and 75% (for communication questions with the SIAP administrator). The details can be seen in Table 2. Taking a Long Time to Access the SIAP Application According to research conducted by several experts about computer-based applica- tions, the average access needed to open ap- plications is 4.7 seconds (Hastomo & Yuhana, 2013), 0.93 seconds (Shubhi, Yuliana, & Winarno, 2011), and 6.959 kB/s (Mulyana & Sholekan, 2010). However, each application used certainly has different access times. Soft- ware specifications used, when the situation opens access (morning, noon, or night) (Mulyana & Sholekan, 2010), availability of Wifi (Pangesti, 2017), as well as the software used are some of the factors that cause the speed of loading or data access. Surely, it also applies to the SIAP appli- cation that is accessed by the writing partici- pants the question that became the respon- dent of this study. Of the 295 respondents who participated, 189 respondents answered 'Yes' and 106 respondents answered 'No'. It means that most respondents need a long time to access SIAP. There were 86 people from all respondents who answered 'No' (45.50%) respond to the next questions. This number 86 consists of 42 female and 44 male respondents. It seems to support the results of the study by Parquette (1952) that men tend to be difficult to understand reading at once reading. Although only two points adrift, it can be said that men need time to under- stand reading compared to women. In fact, in the research conducted by Qian, Buchmann, and Zhang (2018), women tend to have good educational adaptability. However, if they hold the opinions of respondents number 31 and 82 (both women of the same age), in the part of the question regarding the experience of accessing the SIAP application, they wrote that the problem they experienced when accessing was net- work. They will not have trouble as long as the signal when they open the application is in a strong position or from the side of the SIAP application itself, which is in maintenance status. Indeed, there are respondents (number 71, male) who say it takes two minutes at the beginning of the login, but it becomes smooth during the process of writing questions and saving and submitting. For this reason, he did not consider that it took a long time to access the SIAP application. Sometimes, their under- standing of applications that are still relatively new or they are still unfamiliar with using applications makes them stutter technology (based on respondents' answers number 37, 59, and 154; all three are male and are in the same age group, namely 30 to 39 years old). It is undeniable that most people need a long time to understand or learn something new. The same is the case with the results of a re- search by Onwuegbuzie et al. (2004) conduct- ed on undergraduate graduates in America. Communication with the Person in Charge of the Subject Matter SIAP application provides an opportu- nity for the writer of the questions to commu- nicate with the PIC of the material through the Chat feature. However, some participants did not use the facility. Of the 295 respon- dents, 200 people (68%) answered 'Yes', 92 people (31%) answered 'No', and three people (1%) did not answer. Of the 92 respondents who answered 'No', 30 respondents continued to answer questions. Of the total 92 respon- dents who answered 'No', 30 respondents still answered further questions. Male respondents numbered 13 people (43%) and female re- spondents numbered 17 people (57%). It indi- cates that more females are not willing to communicate with the PIC of the subject matter. This condition is in contrary to consis- tency in continuing to answer questions, namely, in this case, women can be said to be inconsistent, which can be caused by psycho- logical factors (Hornung, 1977), such as stress due to communicating with the PIC of the The respondent factors on the digital questionnaire... Muhardis, Burhanuddin Tola, & Herwindo Hariwibowo 148 - Copyright © 2019, REiD (Research and Evaluation in Education), 5(2), 2019 ISSN 2460-6995 subject matter that must be socially elevated. In addition to stress, personal health condi- tions or family can also become the trigger. For example, the condition that occurred in respondent number 264 (female, aged 40 to 49 years old) said that she could not focus be- cause she was caring for his mother who was hospitalized. The same is true for the answer of respondent number 164 (female, aged 50 years old or more) who said that she accessed the SIAP application when she was not ready because she was ill. It is possible that female respondents did communicate with the PIC of the subject matter, but they received a rejection (it could be in the form of neglecting chat, having to make improvements to the questions submit- ted, even more extreme: the rejection of the question). It seems that in this case, the sti- mulus factors and responses (Hovland, 1948) are the determinants of communication that can be established well or not. The reason for fear of being rejected for the questions they wrote seems to be justified when reading the respondent's answers in the question section on the experience of accessing the SIAP application. Respondent number 238, female, postgraduate education, with five years teach- ing experience (respondents included in the sample who answered follow-up questions) wrote that she was happy and worried about the questions she wrote and submitted. She was afraid that the problem would be rejected so she was very aware of writing questions and submitting them. Errors in running an application can also cause unauthorized writ- ten questions. These errors include mistakes in placing stimulus, choice of answers, and key answers and reasons. Subconsciously, those female respon- dents accidentally choose the option 'No'. It is likely the inability of females to suppress ex- pressive language tendencies (Maynard, 1988) which is more dominant than male language behavior. She means to answer 'Yes', but acci- dentally clicks the choice of 'No'. Communication with the SIAP Administrator The results of the studies have provided information that technological advances in the field of communication cause various changes in society, such as changes in the speed of in- formation exchange (Zamroni, 2009). Admin- istrators as parties that facilitate the questions writers with various information, such as, the time of the assignment, indicators that must be done, the number of questions that must be written, must be able to take advantage of technological advances in communicating. To support this, in addition to providing commu- nication facilities with the PIC of the material directly, the SIAP application also provides communication facilities between the question writer and the SIAP administrator. Of the 295 respondents who respond- ed, 266 people (90.16%) used this facility. The number of male respondents was 112 people and female respondents amounted to 154 people, while the number of respondents who answered 'No' was four people and those who did not answer are 25 people. Of the four respondents who answered 'No', only three people participated in answering further ques- tions. The interesting thing is from the three questions that have follow-up questions, spe- cifically the communication questions with the admin getting the response that did not answer the most, namely 25 people. There were 13 female respondents with undergrad- uate education and five male students; four females with post-graduate education and three males with post-graduate education. The biggest possibility of respondents who did not provide answers regarding com- munication with the SIAP administrator was respondents who had less memorable experi- ences. For example, the answer of respondent number 285 (female, 50 years old or more, post-graduate education) on the question of the criticism and suggestions that she request- ed that the SIAP administrator be faster and more responsive in answering the questions submitted by the participants. They need im- mediate answers because the time given by the system for writing questions is not too long, especially for those who communicate via WhatsApp private lines (respondent num- ber 225, woman, undergraduate). Another reason identified was the re- spondents' unpreparedness when they got a call from the SIAP administrator who tells them about assigning questions. Often the ad- The respondent factors on the digital questionnaire... Muhardis, Burhanuddin Tola, & Herwindo Hariwibowo Copyright © 2019, REiD (Research and Evaluation in Education), 5(2), 2019 - 149 ISSN 2460-6995 ministrator suddenly contacts the question writer to be assigned several indicators that must be written for a short period of time (re- spondent's answers number 66, 198, 202, and 290). Upset? Most people will feel annoyed if they have just gotten information but have been demanded to be able to immediately implement with deadlines that are not so long. As an administrator, it is appropriate to provide the best service to service recipients. Everyone who serves as an administrator must have high commitment and motivation in carrying out their duties (Moenir, 2010). If the administrator can provide optimal service and meet customer demands, the organization can be regarded as the greatest form of suc- cess on the service side. On the contrary, if the service provided is not satisfactory, it is necessary to improve the system and imple- mentation mechanism. Conclusion The web-based questionnaire or digital questionnaire still has many disadvantages. The presence of data outliers, for example, cannot be avoided. There are only factors that cause respondents to give unexpected data, such as psychological factors when respond- ing to questionnaires. Actually, it can be anti- cipated if the digital questionnaire is accom- panied by an interview to get in-depth infor- mation about the aspects being evaluated. It is okay to use a web-based questionnaire, but in its implementation, it will be more targeted if the respondent is accompanied when input- ting data on the media used (such as devices, mobile phones, tablet computers, etc.). It pro- vides an opportunity for respondents to ask if there are questions that are difficult to under- stand. Interviewers can also supervise careful- ly, especially taking preventive actions related to conditional questions (in this paper stated in terms of follow-up questions). Data in the form of outliers do not have to be deleted or treated by means of statistical averaging, but can be used as a data source to conduct in-depth interviews with re- spondents who gave the response. It can be a new field for constructivist paradigm re- searchers. They can explain outliers from an- other point of view. Even for psychometrics, outliers can be used as new data to see the relationships between the variables of non- parametric scale data found in the respon- dents, for example, to see if there is a rela- tionship between the length of teaching time, gender, and age with communication with the subject matter PIC, and SIAP administrator. In addition, it is suggested that in the formulation of questionnaires (both paper- based and web-based), it is necessary to con- sider the respondent's experience of novelty and current, such as responding in a new form (digital questionnaire), layered questions (questions that require further questions), because not all of the respondents were in a focus position to answer, even though they were highly educated, of productive age, and had a longer experience. References Ali, M. A. (1994). Identifying the location shift outliers that matter. Sankhyā: The Indian Journal of Statistics, Series A, 56(3), 500–511. Retrieved from https:// www.jstor.org/stable/25051015 Cheng, Y. H., & Tsai, C. C. (2017). Online research behaviors of engineering graduate students in Taiwan. Journal of Educational Technology & Society, 20(1), 169–179. Retrieved from https:// scholar.lib.ntnu.edu.tw/en/publications /online-research-behaviors-of-engineer ing-graduate-students-in-tai Fíncher, C. (1981). AIR between forums: The literature of program evaluation. Research in Higher Education, 14(3), 277– 280. Retrieved from https:// www.jstor.org/stable/40195416 Harris, H. (2001). Content analysis of secondary data: A study of courage in managerial decision making. Journal of Business Ethics, 34, 191–208. https:// doi.org/10.1023/A:1012534014727 Hastomo, F., & Yuhana, U. L. (2013). Perancangan dan pembuatan perangkat lunak aplikasi Android untuk pengolahan data transaksi pada perusahaan telekomunikasi “X” dengan menggunakan Pentaho. Jurnal Teknik The respondent factors on the digital questionnaire... Muhardis, Burhanuddin Tola, & Herwindo Hariwibowo 150 - Copyright © 2019, REiD (Research and Evaluation in Education), 5(2), 2019 ISSN 2460-6995 POMITS, 2(1), 77–82. Retrieved from http://ejurnal.its.ac.id/index.php/tekni k/article/view/2733 Hornung, C. A. (1977). Social status, status inconsistency and psychological stress. American Sociological Review, 42(4), 623– 638. https://doi.org/10.2307/2094560 Hovland, C. I. (1948). Social communication. Proceedings of the American Philosophical Society, 92(5), 371–375. Retrieved from https://www.jstor.org/stable/3143048 Martel, A. R. (2015). The detection of outliers in nondestructive integrations with the Generalized Extreme Studentized Deviate test. Publications of the Astronomical Society of the Pacific, 127, 258–265. Retrieved from https:// iopscience.iop.org/article/10.1086/680 382/meta Maynard, D. W. (1988). Language, interaction, and social problems. Social Problems, 35(4), 311–334. https://doi.org/ 10.2307/800590 McArthur, D. L., & Choppin, B. H. (1984). Computerized diagnostic testing. Journal of Educational Measurement, 21(4), 391– 397. https://doi.org/10.1111/j.1745- 3984.1984.tb01042.x McQuarrie, A. D., & Tsai, C.-L. (2003). Outlier detections in autoregressive models. Journal of Computational and Graphical Statistics, 12(2), 450–471. Retrieved from https://www.jstor.org/ stable/1391204 Moenir, A. S. (2010). Manajemen pelayanan umum di Indonesia. Jakarta: Bumi Aksara. Mulyana, A., & Sholekan, S. (2010). Aplikasi mini market online berbasis web. Bandung: Universitas Telkom. Nezami, E., & Butcher, J. N. (2000). Objective personality assessment. In G. Goldstein & M. Hersen (Eds.), Handbook of Psychological Assessment (3rd ed., pp. 413–435). https://doi.org/ 10.1016/B978-008043645-6/50094-X Nulty, D. D. (2008). The adequacy of response rates to online and paper surveys: What can be done? Assessment & Evaluation in Higher Education, 33(3), 301–314. https://doi.org/10.1080/ 02602930701293231 Onwuegbuzie, A. J., Mayes, E., Arthur, L., Johnson, J., Robinson, V., Ashe, S., … Collins, K. M. T. (2004). Reading comprehension among African American graduate students. The Journal of Negro Education, 73(4), 443–457. https://doi.org/10.2307/4129628 Pangesti, B. N. A. (2017). Analisa kecepatan transfer data pada perancangan hotspot sederhana dengan system single sign on di perkantoran. Fountain of Informatics Journal, 2(1), 1–7. https://doi.org/ 10.21111/fij.v2i1.814 Parquette, W. S. (1952). Intensive reading. The English Journal, 41(2), 78–82. https:// doi.org/10.2307/809208 Prescott, P. (1978). Examination of the behaviour of tests for outliers when more than one outlier is present. Journal of the Royal Statistical Society, Series C (Applied Statistics), 27(1), 10–25. Retrieved from https://www.jstor.org/ stable/2346221 Qian, Y., Buchmann, C., & Zhang, Z. (2018). Gender differences in educational adaptation of immigrant-origin youth in the United States. Demographic Research, 38, 1155–1188. https://doi.org/ 10.4054/DemRes.2018.38.39 Rapallo, F. (2012). Outliers and patterns of outliers in contingency tables with algebraic statistics. Scandinavian Journal of Statistics, 39(4), 784–797. https://doi. org/10.1111/j.1467-9469.2012.00790.x Reja, U., Manfreda, K. L., Hlebec, V., & Vehovar, V. (2003). Open-ended vs. close-ended questions in web questionnaires. Developments in Applied Statistics / Metodološki Zvezki, 19, 159– 177. Ross, L., & Cronbach, L. J. (1976). Handbook of evaluation research. Educational Researcher, 5(10), 9–19. https://doi.org/ 10.3102/0013189X005010009 The respondent factors on the digital questionnaire... Muhardis, Burhanuddin Tola, & Herwindo Hariwibowo Copyright © 2019, REiD (Research and Evaluation in Education), 5(2), 2019 - 151 ISSN 2460-6995 Sevillano-García, M. L., & Vázquez-Cano, E. (2015). The impact of digital mobile devices in higher education. Educational Technology & Society, 18(1), 106–118. Shubhi, M. M. F., Yuliana, M., & Winarno, I. (2011). Sistem monitoring jaringan menggunakan BREW (Binary Runtime Environment for Wireless). Retrieved from http://repo.pens.ac.id/1077/1/Paper_ Sidang_TA.pdf SIAP Application Development Questionnaire. (n.d.). No Title. Retrieved from https://docs.google.com/forms/ d/e/1FAIpQLSemVx0QSQlom5UecJs 55aVE2N5kHMcRJSQZTdXHp4EGc T-N-A/viewform Yanxia, Y. (2017). Test anxiety analysis of Chinese college students in computer- based spoken English test. Journal of Educational Technology & Society, 20(2), 63–73. Retrieved from https://www. jstor.org/stable/90002164 Zamroni, M. (2009). Perkembangan teknologi komunikasi dan dampaknya terhadap kehidupan. Jurnal Dakwah: Media Komunikasi Dan Dakwah, 10(2), 195– 211. https://doi.org/10.14421/jd.2009. 10205