JOURNAL OF FOREIGN LANGUAGE TEACHING & LEARNING Vol.2 No. 2 July 2017 ABSTRACT This study aims to design a survey instrument that can be used to collect information on the relationships between the ICT-related learning experiences of the English language pre-service teachers in Yogyakarta, Indonesia, and their technological pedagogical content knowledge (TPACK). Qualitative and quantitative research methods were used to analyse the degree of the reliability and validity of the instrument. The result suggests that this instrument meets the general require- ments to be used in a larger scale of work in investigating the role of pre-service teachers’ experiences in learning to use ICT in their pedagogical practice in influencing the development of their TPACK. Keywords: learning experience; technological pedagogical content knowledge (TPACK); validity; reliability INTRODUCTION The purpose of this study is to develop an instrument that can be used to exam- ine the relationship between the technology-related learning experiences of the English language pre-service teachers at a teacher training institution in Yogyakarta, Indonesia, and their current level of Technological Pedagogical Content Knowl- edge (TPACK). TPACK is a current framework which emerged as a response to- ward the ineffectiveness of Information and Communication Technology (ICT) to influence educational improvement and student learning achievement. Suc- cessful ICT integration in learning and teaching consider technology not as an The Development of the Survey of Technology Use, Teaching, and Technology- Related Learning Experi- ences among Pre-Service English Language Teachers in Indonesia Dyah Setyowati Ciptaningrum c ompleted her study majorin g in I CT in Edu - cation in Monash University, Australia, in 2007, an d rec eiv ed h er Ed.D degree from Flin ders Univ ersity, Austr alia, in 2015. She is currently wor king at Yogyakar ta S tate Univer sity, In do- nesia, as a lecturer in English language edu- c ation depar tmen t. Her r esearc h in ter est in - c lu des th e u se of I CT in edu c ati on , En glish lan gu age learn in g and teac h in g, an d teac her profession al learn in g. 11-26 12 Journal of Foreign Language Teaching & Learning Vol.2 No.2 July 2017 end in itself but it needs to be related to the con- tent of school subject, good pedagogy, and class- room context (Mishra & Koehler, 2006). This study is important within the recent con- text of education in Indonesia. The Indonesian Ministry of Education (MoNE) has mentioned that Indonesian teachers need to integrate ICT in the learning and teaching process (Ministry of National Education, 2007a; Ministry of National Education, 2007b; Ministry of National Education, 2009). To support the ICT integration MoNE has invested in the provision of ICT infrastructure in schools (Ministry of National Education, 2010) by provid- ing schools with computers, Internet connection and online learning content (p. 28, 31). MoNE has also invested in various ICT-related teacher profes- sional developments (The United Nations Educa- tional, Scientific and Cultural Organization [UNESCO], 2007; Belawati, 2005). ICT has the potential to contribute to the im- provement of Indonesian students’ English lan- guage proficiency. The Internet has made access to authentic materials, vast linguistic resources and an exhaustive range of materials in all languages easier. Thanks to the Web 2.0 technology, teach- ers and students of languages are able to commu- nicate with each other across the globe. With ICT, learning languages is no longer confined within school walls. Students’ preferred learning styles can also be catered for by the use of ICT. However, this potential of ICT will be realized if teachers’ use of ICT in the classroom is guided by principles of good curriculum design and pedagogy for teach- ing English. Within this context, the role of pre-service teacher education becomes crucial as it serves as the initial and primary source of teachers’ knowl- edge. Putnam and Borko (2000) argue that “How a person learns a particular set of knowledge and skills, and the situation in which a person learns, become a fundamental part of what is learned” (p. 4). What teachers learned during their pre-service study would influence the way they teach as in-ser- vice teachers. Teachers’ knowledge base needs to be expanded to include knowledge of ICT use in education that is closely connected with curriculum and good pedagogy. TPACK has become the frame- work for restructuring teacher education programs in preparing teachers to teach with technology. There have been a number of studies that de- velop instruments to measure the teachers’ TPACK (Koehler & Mishra, 2005; Koh, Chai, & Tsai, 2010; Sahin, 2011; Schmidt et al., 2009/2010). Koehler and Mishra (2005) conducted a survey to assess the impact of a certain course on educational technol- ogy in influencing the participants’ perception of their understanding of content, pedagogy, and technology. Thus, this instrument is subject-specific. Schmidt et al. (2009/2010) designed a survey that measured teachers’ understanding of each compo- nent of TPACK. Even though they claim that their survey was designed for general contexts and mul- tiple content areas (p. 128), this survey is still con- tent and context specific as it is designed to be used by K-12 pre-service teachers in the U. S. who are prepared to teach science, mathematics, social stud- ies, and literacy. However, the items within each of these subjects are noticeably similar while there are differences in the content and pedagogy of each subject. Sahin (2011) also developed a TPACK sur- vey for more general use. His survey is intended to measure the TPACK of pre-service teachers regard- less of their major. Koh, Chai, and Tsai’s (2010) instrument was designed for general use as well but 13 Journal of Foreign Language Teaching & Learning Vol.2 No.2 July 2017 within Singapore educational contexts. Since the TPACK framework itself indicates that the effec- tive use of technology has to be context-specific, the instrument needs to be specifically developed for a particular school subject within the unique classroom context surrounding the teaching of that subject. Teacher knowledge is influenced by their learn- ing experience. Research on effective teacher pro- fessional development (PD) suggests that ICT-re- lated teacher PD should value teachers as adult learners and be conducted in a constructivist in- structional approach to facilitate meaningful learn- ing (Hawley and Valli, 1999; Garet, Porter, Desimone, Birman, & Yoon, 2001; Desimone, 2009). Most importantly, ICT-related teacher PD needs to be seen as a systematic effort by taking into consideration teachers’ contextual factors in the PD design to influence changes in teachers’ classroom practices to enhance student learning (Guskey, 2000; Desimone, 2009). The existing survey instruments on TPACK were designed for the educational context of the West- ern, developed countries that have different socio- cultural factors from Indonesia and they did not attempt to tap teachers’ perceptions on their ICT- related learning experience. Besides, there is a lack of data on how the principles of quality ICT-related teacher PD work in the Indonesian educational context. Therefore, it is important to design an instrument that can measure the level of TPACK of Indonesian pre-service teachers and their per- ceptions concerning the quality of their ICT-related learning experiences. Considering the existing instruments are usu- ally written for school subjects such as Math, Sci- ence, and Social Studies, the present study modi- fies the work of Schmidt et al. (2009/2010) and Sahin (2011) on the TPACK survey by incorporat- ing Indonesian English language teachers’ pedagogi- cal content knowledge into the teachers’ TPACK measurement instruments. Since the TPACK level of Indonesian EFL pre-service teachers and their learning experience that shape the current devel- opment of their TPACK have not been studied yet, this study attempts to bridge this gap. Thus, the question addressed in this study is whether the sur- vey instrument developed in this study valid and reliable to measure the TPACK levels of the En- glish language pre-service teachers at a teacher train- ing institution in Yogyakarta, Indonesia. The questionnaire may become a basis in evalu- ating the outcome of pre-service education institu- tions in Indonesia, particularly their graduates’ readiness to use ICT in their pedagogical practices. The questionnaire may also be useful to inform the development of effective interventions to as- sist the Indonesian English language pre-service teachers in developing their TPACK. FOREIGN LANGUAGE TEACHERS’ TPACK The idea of TPACK has been built on Shulman’s notion of pedagogical content knowledge (1986, 1987). Shulman (1987), as cited in Mishra and Koehler (2006), argues that teacher’s knowledge consists of “content knowledge, general pedagogi- cal knowledge, curriculum knowledge, pedagogical content knowledge, knowledge of learners and their characteristics, knowledge of educational con- texts and knowledge of educational ends, purposes, and values, and their philosophical and historical grounds” (p. 8). He went further by stating that content knowledge and pedagogical knowledge in- tersected in the minds of the teachers (Figure 1); 14 Journal of Foreign Language Teaching & Learning Vol.2 No.2 July 2017 thus, making the pedagogical content knowledge (PCK) central in the body of knowledge of teach- ing. Mishra and Koehler (2006) propose a framework that includes the integration of technological knowledge into the pedagogical content knowledge. They stated that in order to realize the potential of ICT in the teaching and learning process, teach- ers needed to develop a knowledge that showed a connection and interaction among technological knowledge, content knowledge and pedagogical knowledge (Figure 1). In addition to Shulman’s categorization of teacher’s knowledge, Mishra & Koehler’s framework yields to the development of technology knowledge, technological content knowledge, technological pedagogical knowledge and technological pedagogical content knowledge. Technology Knowledge (TK) refers to the skills to use the technology. Teachers need to show the ability to use the standard technology like the black/white board, textbooks, visual aids, or the new technology like the Internet and digital video. Including in this knowledge are teachers’ skills to operate computer system and hardware, and use software tools like word processors, PowerPoint, spreadsheet, web browsers, e-mail, and instant messaging. Digital technology is continuously changing. It is imperative for teachers to have the ability to keep up and adapt with the changes in technology. In addition, teachers should also need to decide whether the technology supports or hin- ders the attainment of the purpose of the lesson (Mishra & Koehler, 2008). FIGURE 1: TECHNOLOGICAL PEDAGOGICAL CONTENT KNOWLEDGE AS A RESULT OF THE BLENDING OF TECHNOLOGY KNOWLEDGE, CONTENT KNOWLEDGE AND PEDAGOGICAL KNOWLEDGE (KOEHLER & MISHRA, 2008, P. 12). Technological Content Knowledge (TCK) in- cludes the ability to select the appropriate technol- ogy tool to deliver the subject matter since tech- nology can support or impede the learning of the subject matter. The nature of the ideas in the sub- ject matter drives the selection process. This is a combination of content knowledge and technol- ogy knowledge. Richards (1998), as cited in van Olphen (2008), argues that language teachers’ con- tent knowledge includes an understanding of lin- guistics components (phonetics, phonology, mor- phology, semantics, syntax, socio-linguistics, prag- matics), second language acquisition, cross-cultural awareness, and the development of language pro- ficiency skills (reading, writing, speaking, and lis- tening). TCK for foreign language teachers can be defined as “the body of knowledge that teachers have about their target language and its culture and how technology is used to represent this knowl- edge” (van Olphen, 2008, p. 113). Technological Pedagogical Knowledge (TPK) is the interaction between technology and pedagogy. Teachers have a repertoire of teaching strategies and they should be able to skillfully select the one that best represents the idea in the subject matter 15 Journal of Foreign Language Teaching & Learning Vol.2 No.2 July 2017 and suits the students’ context or characteristics such as age, fluency/mastery level of the topic, learning style, or background knowledge. With technology, the complexity increases. Teachers need to under- stand how technology can change the teaching and learning. There are different technology tools that can be used for a task. The selection of the appro- priate tool is “based on its fitness, strategies for using the tool’s affordances, and knowledge of pedagogical strategies and the ability to apply those strategies for use of tech- nologies. This includes knowledge of tools for main- taining class records, attendance, and grading, and knowledge of generic technology-based ideas such as WebQuests, discussion boards, and chat rooms” (Mishra & Koehler, 2006, p. 1028). Technological Pedagogical Content Knowledge (TPACK) is the heart of effective teaching using technology. It requires “an understanding of how to represent concepts with technologies, pedagogical techniques that use tech- nologies in constructive ways to teach content; knowl- edge of what makes concepts difficult or easy to learn and how technology can help students learn; knowl- edge of students’ prior knowledge and theories of epis- temology; and knowledge of how technologies can be used to build on existing knowledge and to develop new epistemologies or strengthen old ones” (Mishra & Koehler, 2008, p. 10). According to the American Council of Teach- ers of Foreign Languages (ACTFL, 2002) Program Standards for the Preparation of Foreign Language Teachers, the knowledge that foreign language teachers should be able to demonstrate consists of the following six content standards: (1) language, linguistics, comparisons; (2) Cultures, literatures, cross-disciplinary concepts; (3) Language acquisition theories and instructional practices; (4) Integration of standards into curriculum and instruction; (5) Assessment of languages and cultures; (6) Profes- sionalism. The Teacher of English to Speakers of Other Languages (TESOL) also released a docu- ment containing a set of standards that need to be made in preparing foreign language teachers. Briefly, teacher candidates are expected to show proficiency in the following five domains, each is divided into a number of standards: (1) language; (2) culture; (3) instruction; (4) assessment; (5) pro- fessionalism. Explanations, rubrics, and perfor- mance indicators of the standards and domains are provided in these two documents. The knowledge that is covered in these documents incorporate the notion of pedagogical content knowledge proposed by Shulman (1986; 1987). Using Mishra & Koehler’s concept of TPACK, van Olphen (2008, p. 117) states that meaningful technology integration in language teaching entails the following condition: a) An understanding of how linguistic and cultural concepts can be represented using technology b) Educational approaches to language teaching that draw from socio-constructivist philosophies to develop students’ language and cultural com- petence c) An awareness of what facilitates or hinders the acquisition of language and the development of language competence and how technology, specifically CALL or CMC, can revamp com- mon problems that students ordinarily face d) An awareness of students’ previous knowledge, and particularly knowledge of second language acquisition and cognitive development theories 16 Journal of Foreign Language Teaching & Learning Vol.2 No.2 July 2017 e) An understanding of how current and emerg- ing technologies can be used to advance present knowledge and to develop new epistemologies and sustain previous ones. QUALITY LEARNING IN DEVELOPING FOR- EIGN LANGUAGE TEACHERS’ TPACK Learning for teachers is an ongoing and con- tinuous process which also includes activities that are embedded in their daily lives (Desimone, 2009). Ref lecting, reading journal or magazine, group discussion, teacher network or study group, self- or observer examination of the teachers’ practice, teachers’ individual activities, such as engagement in educative online venues are examples of teacher learning activities (Desimone, 2009). Thus, there are different forms of learning that can be per- formed by teachers to improve their knowledge on ICT integration. Technology related teacher professional development shows a movement from one-size-fits-all type of training or workshops that focus on showing teachers how to use the technol- ogy hardware and software (Denning & Selinger, 1999) to those that are conducted over time with the element of follow-up learning and feedback (Cole, Simkins & Penuel, 2002; Kariuki, Franklin, & Duran, 2001; Mulqueen, 2001). Studies on teachers’ learning should focus on the critical features of teachers’ learning experi- ences (Desimone, 2009). Several studies (Campbell, McNamara, and Gilroy, 2004; Garet, Porter, Desimone, Birman, & Yoon, 2001) conclude that teachers’ learning models can impact student achievement if they have the following features: 1. longer in duration in terms of contact hours plus follow-up in order to be sustainable 2. actively engage teachers in meaningful and rel- evant activities for their individual contexts 3. school-based 4. provide a degree of autonomy for teachers to design and choose the topics and types of PD that suit their need and contexts 5. promote peer collaboration and community building 6. have a clear goal toward student achievement 7. provide access to new technologies for teaching and learning TPACK framework has been used recently to underline models of professional development. Learning-by-design approach is an example where the TPACK framework and the critical features of teacher learning are used. In this model of teacher learning, teachers need to construct artifacts (such as online courses, digital video, podcasts, and so on) based on the content of the subjects taught by the teachers to be used in their own classroom (Angeli & Valanides, 2009; Beckett et al., 2003; Cole, Simkins & Penuel, 2002; Keller, Hixon, Bonk, & Ehman, 2004; Koehler, Mishra, & Yahya, 2007; Mulqueen, 2001). Koehler and Mishra (2005) mention that learning by design approach focuses teachers’ attention on a problem they might en- counter in thei r practi ce; then the y wo rk collaboratively with other participants to investi- gate the ways in which technology can be used to address the problem. This approach is informed by the principles of social constructivism or con- structionism with the participants actively construct their knowledge on a particular topic with the help of their peers by creating artifacts that meet their teaching goals. Design projects lead to sustained inquiry and revision of ideas (Koehler & Mishra, 2005). Learning in this kind of environment hap- 17 Journal of Foreign Language Teaching & Learning Vol.2 No.2 July 2017 pens informally and within the immediate context of the participants which results in deeper under- standing of the topic. Problem-based learning also influences this approach since the length of the program is extended than the traditional one-shot type of training, the activities to solve the ‘real- world’ problems are learner centered, interdisci- plinary, and ‘ill-structured’ where there can be more than one solution to the problem (Koehler & Mishra, 2005). This kind of learning environment required a pedagogical shift on the role of the learn- ers and the teacher/instructor. The learners have to be like an ‘apprentice’ who investigate the prob- lem and find solutions with the help of their peers (who might have more or less knowledge on the topic under investigation) in the actual context of practice. The teachers/instructors assist learners to understand the content, provide them with feed- back, mentor and coach, and manage the learn- ing context and setting. They no longer become the main source of information who transmit their knowledge to their students. Hence, learning by design approach reflects the principles of transformational adult learning. It allows the participants to exercise self-directedness (Brookfield, 1991), provides more learners’ engage- ment, and builds connections with their real need and context (Eraut, 2007; Borko, 2004). There are also opportunities to critically reflect on their ex- periences in learning and teaching as well as build- ing a learning community. The whole process re- sults in the ownership of the program, a sense of agency. This kind of learning environment creates meaningful learning experiences that will highly likely make the learning sustained even after the program has finished (Lawless & Pellegrino, 2007). METHOD The purpose of this study is to develop an in- strument that can be used to examine the relation- ship between the technology-related learning ex- periences of the English language pre-service teach- ers at a teacher training institution in Yogyakarta, Indonesia, and their current level of Technologi- cal Pedagogical Content Knowledge (TPACK). In line with this purpose, the Survey of Technology Use, Teaching, and Technology-Related Learning Experiences among Pre-Service English Language Teachers was constructed. Survey design is the appropriate method under- lying this study. According to Creswell (2011), sur- vey research design is a quantitative research pro- cedure where a sample or the entire population of people complete a set of questions (questionnaire) to describe the opinions, attitude, behaviours, or characteristics of the population. In order to inves- tigate the validity and reliability of this instrument, it needs to be tested by sending the instrument to a sample of English language pre-service teachers in Yogyakarta, Indonesia and asking them to com- plete it. Since the population of English language pre-service teachers in Yogyakarta is quite large and geographically dispersed, survey design enables this study to collect information from a few respondents to describe the characteristics of the whole popula- tion, which is cost effective and time efficient (Salant & Dillman, 1994). Since survey design does not rely on observa- tion and long, structured or semi-structured inter- view that utilise open-ended questions to collect data, survey design cannot provide the depth of understanding that interview and observational techniques provide (Salant & Dillman, 1994). In order to address this issue, the instrument designed 18 Journal of Foreign Language Teaching & Learning Vol.2 No.2 July 2017 in this study included two essay (open-ended) ques- tions and two semi-closed-ended questions to elicit qualitative information from the respondents. INSTRUMENT DEVELOPMENT Reviewing the literature around the existing surveys used to measure teachers’ TPACK was the first step conducted in the development of the in- strument in this study. The instrument used was adapted from Schmidt et al. (2009/2010) and Sahin (2011) to measure Technological Pedagogical Content Knowledge of Indonesian English language pre-service teachers at a teacher training institution in Indonesia. This study’s instrument focused on the specific content and pedagogical knowledge related to learning and teaching foreign language, i.e. the English language. The literature around teacher learning was also consulted in order to develop the items about the ICT-related learning experiences of the English language pre-service teachers. There are five domains in the questionnaire. Four domains measure TPACK perceptions on Technological Knowledge (TK), Technological Con- tent Knowledge (TCK), Technological Pedagogical Knowledge (TPK), and Technological Pedagogical Content Knowledge (TPACK). One domain mea- sures the pre-service teachers’ perceptions on their ICT-related learning experiences. Demographic questions are included to identify the characteris- tics of the respondents in order to understand gen- der differences or relationships between teachers who have access to technologies at home and those who do not. The TK domain collects information on English language pre-service teachers’ skills in operating technological hardware and software, which are generally available in the context of these teach- ers. The TCK domain covers questions about the teachers’ use of technology in enhancing their knowledge on the non-teaching topics they have enrolled at the English language and education study program. The TPK domain aims to collect information on the teachers’ use of technology to improve their knowledge and skills in teaching. The TPACK domain contains questions about the interrelationship among technology, content and pedagogical knowledge that influence the teach- ers’ English language and teaching skills. The ques- tions in the ICT-related learning experiences do- main are designed to collect information on the teachers’ perceptions on their learning experiences that might inform their level of TPACK. This questionnaire uses multiple types of ques- tions and response formats which are carefully con- structed to minimize common responses or com- mon method variance which can cause measure- ment error and mislead conclusions (Podsakoff, MacKenzie, & Lee, 2003). Unlike the instruments designed by Schmidt et al. (2009/2010) and Sahin (2011) where they used the same question and re- sponse format which raise an issue concerning ‘con- sistency motif’ of the respondents (Podsakoff, MacKenzie, & Lee, 2003), this questionnaire also incorporates different types of questions that re- quire the use of different response formats. Initially, there was a total of 64 items in this instrument. Most of the items (36 items) used five- point Likert-type response scale ranging from ‘strongly disagree’ to ‘strongly agree’ with the in- clusion of ‘neutral’ option. 18 other items were also based on five-point Likert-type scale, but the op- tions were labelled differently (from ‘very compe- tent’ to ‘not competent’ with the addition of ‘not 19 Journal of Foreign Language Teaching & Learning Vol.2 No.2 July 2017 applicable’ option). Research surrounding the num- ber of options in response scale has been inconclu- sive (Lietz, 2008). For example, Nagata, Ido, Shimizu, Misao, and Matsuura study’s (1996) showed that the 5-point scale was the easiest of the other types of response scales to complete when applied to instruments for assessing health status. Finn and Peng’s study (2009), however, showed that seven category responses outperformed five category re- sponses for both Likert and semantic differential item formats when scaling marketing stimuli. Cook, Cella, Boespflug, and Amtmann (2010) ar- gued that four to five response categories were bet- ter than two to three. However, their study also found that more than five categories did not nec- essarily improve the reliability, person separation, or validity of scores. Thus, five-point response cat- egories were adopted in the initial development of this study’s questionnaire on TPACK and technol- ogy related learning experiences among pre-service English language teachers in Indonesia. In the ques- tionnaires, two items adopt ordering and ranking type of question, two items are written in multiple choice/selection, and 1 item is written in open- ended question. Respondents was also informed that their answers would be anonymous, there were no right or wrong answer, and their answers would not be used for any marking purposes to reduce ‘mood state’ effect (Podsakoff, MacKenzie, & Lee, 2003). RESEARCH SITE AND PARTICIPANTS The sur vey was crea ted online b y using SurveyGizmo 14-day trial program. The link of this survey was sent to 133 English language pre-ser- vice teachers of a teacher training institution in Yogyakarta, Indonesia, who were listed on the researcher’s Facebook friend list. The 133 pre-ser- vice teachers made up this study’s target popula- tion. Their response indicated their informed con- sent. The first reminder to participate on the sur- vey was sent two days after the survey was launched, followed by the second reminder two days later. The reminders were posted on the researcher’s Facebook wall and sent to the participants’ inbox messages. Thirty-seven responses were received. Out of this number, fifteen responses were partial (incomplete). A number of respondents sent the researcher personal messages through Facebook regarding technical problems they encountered when trying to complete the survey. It appeared that some of the respondents were not familiar with this kind of online survey and stopped com- pleting the survey after they clicked the first ‘next’ button, which explained the high occurrence of partial responses. Thus, there were only 22 respon- dents who were selected as the sample of this study. As for the language that was used in the question- naire, it was decided to use English since the re- spondents of this study are pre-service English lan- guage teachers who understand English well. The procedure of the survey development in this study is illustrated in Figure 2. Literature review Research questions Survey questions development Cognitive interview Pilot test Validity and reliability check Add/delete/ change questions FIGURE 2: THE MODEL OF SURVEY PROCEDURE OF THIS STUDY DELIMITATION AND LIMITATION OF THE STUDY In order to provide a good estimate of the popu- lation characteristics, there are several factors that 20 Journal of Foreign Language Teaching & Learning Vol.2 No.2 July 2017 need to be considered in conducting survey re- search (Salant & Dillman, 1994; Creswell, 2011). The number of sample needs to be as large as pos- sible to ensure that the sample represents the tar- get population. Every member of the population also has the same chance of being selected for the sample. The non-responsive respondents in the sample should have similar characteristics with the people who give responses in the sample. The in- strument needs to be well-constructed to avoid any ambiguity both in the questions and in the re- sponses and rigorous administration procedure needs to be implemented to obtain as large a re- turn rate as possible. Due to the limited time un- der which this study needs to be completed, rigor- ous sampling technique is not possible. It is the instrument development that is empha- sized in this study. Expert review as an evidence of validity was unlikely to be conducted due to the funding limitation of this study. Thus, the effort to achieve a degree of validity and reliability was performed by implementing cognitive interview- ing procedure (Desimone & Le Floch, 2004) and by carrying out statistical tests on the responses (i.e. Cronbach Alpha and Factor analysis). To achieve stronger reliability and validity, the initial survey items of this study were modified by the deletion of several items based on the result of the validity and reliability tests. According to Field (2009, p. 681), a second run of factor analysis is essential if the survey items undergoes a number of changes as a result of the statistical tests. With the limited scope of the paper, a second run of factor analysis was not conducted. Moreover, the limited sample size of this study made the application of factor analysis to the whole items not viable. DATA ANALYSIS Qualitative and quantitative research methods were used to analyse the degree of the reliability and validity of the instrument. A cognitive inter- view was applied after the first construction of the survey items. The internal consistency of each do- main in this instrument was analysed by using Cronbach’s alpha reliability technique. Factor analysis was implemented to examine the construct validity of each domain. The two essay (open-ended) questions and two semi-closed-ended questions were not included in this analysis. RESULTS ON THE COGNITIVE INTERVIEWS After the initial survey was completed, cogni- tive interviews were conducted to 5 participants. Cognitive interviews is a method to contribute to increase reliability and validity of surveys (Desimone & Le Floch, 2004). Based on the feedback gath- ered during the cognitive interviews, some items were revised (refer to Appendix 1 for the cognitive interview results). The revision included the fol- lowing: 1) Removal of negative items, which were modi- fied into positive statements, 2) Removal of the adjective ‘appropriate,’ 3) Addition of information to clarify meaning of the statements, such as ‘school work’ instead of ‘work’ only and an example of ‘difficult concept in English language,’ 4) Removal of examples from some statements in TK section to avoid double barrel statement, 5) Emphasis on the instruction of certain items (e.g. the ranking-type question) by formatting the sentence in the instruction with italic, bold, and colour, 21 Journal of Foreign Language Teaching & Learning Vol.2 No.2 July 2017 6) Simplification on the length of several state- ments, 7) Addition of information to make the meaning of the statement clear (e.g. from ‘I do not know how to use technology to assess students’ per- formance’ into ‘When I teach later, I will know how to use technologies to assess students’ per- formance’), 8) Change one of the ranking-type items to a semi- closed-ended type item, 9) Addition of one open-ended item, and 10)Removal of the neutral option from the re- sponse scale. Research on the omission and inclusion of neu- tral option has been inconclusive (Lietz, 2008). The decision to remove neutral option from the re- sponse scale in this survey was based on the result of the cognitive interview which appeared to sup- port the findings that the introduction of neutral option would attract respondents to select this op- tion when they were not completely sure about their answers (Garland 1991; Kalton et al. 1980; Krosnick & Helic 2000; O’Muircheartaigh 2000; Schumann & Presser 1996, as cited in Lietz, 2004). RESULTS ON THE FACTOR ANALYSIS AND CRONBACH’S ALPHA Survey items need to be checked whether they relate to the construct that the study intended to measure (Field, 2009). Factor analysis is a technique for identifying groups or clusters of variables. Each domain in this survey item was analysed by using factor analysis. After the application of factor analy- sis to validate this survey items, the reliability of the scale was examined using the Cronbach’s Al- pha. TECHNOLOGY KNOWLEDGE DOMAIN The construct of this domain is about teachers’ skills to use technology. The factor analysis on the 22 items representing TK resulted in 7 components underlying this construct. These components may, or may not, relate to genuine sub-components of TK. Special attention was given to the items with factor loadings below 0.40 (Field, 2009). These items are presented in Table 1. The result shows that each of these items has a much bigger factor loading in another component. Having closely examined the items of variable TUTTEA3, TUTTEA5, TUTTEA6, TUTTEA7, TUTTEA 9, AND TUTTEA12, it turned out that these items represent the same concept (i.e. ability in operating technologies). Since there were 22 items in this scale (which represented the answers from the 22 sample of this study), it is suspected that the limited sample of this study may result in the low factor loading of these items. The decision was then made that all items that asked the pre- service teachers’ ability in operating technologies (i.e. TUTTEA1 to TUTTEA17) were dropped since these items had a similarity to TUTTEA20 (‘I play around with different technologies’) which had much greater factor loading (.771). TUTTEEA18 and TUTTEEA19 item were also deleted since they appeared to have resemblance with TUTTEA20 item as well. TECHNOLOGICAL CONTENT KNOWLEDGE (TCK) TCK includes the ability to select the appropri- ate technology tool to deliver the subject matter. It is the relationship between content and technol- ogy. Based on the factor analysis, two components had the eigenvalues over 1 and in combination 22 Journal of Foreign Language Teaching & Learning Vol.2 No.2 July 2017 explained the 64.06% of the variance. It means that the 10 items reflected two constructs. The fac- tor loadings were above.40 for each item (i.e..44 to.83). Thus, all items were retained. TECHNOLOGICAL PEDAGOGICAL KNOWL- EDGE (TPK) The factor analysis extracted 2 components for this domain. Since TPK is the interaction between technology and pedagogy, the 6 items in this do- main may reflect these two concepts (technology and pedagogy). Factor loadings were between.51 to.86. This result showed that the factor loadings were considered as good and accepted. No item was changed or deleted. TECHNOLOGICAL PEDAGOGICAL CONTENT KNOWLEDGE (TPACK) TPACK is where technology, pedagogy, and con- tent merge to create a unique notion of effective teaching using technologies. Only one factor emerged as the underlying construct of this scale based on the factor analysis. The 6 items within this domain were built around one coherent con- struct. The factor loadings were between.64 to.90. All items were then retained. TEACHERS’ TECHNOLOGY-RELATED LEARN- ING EXPERIENCE (TLE) This refers to the quality of learning experiences that can inf luence teachers’ development of TPACK. It is predicted from the literature around effective teacher professional learning that teach- ers with positive or high-quality learning experi- ences will have a higher level of TPACK and teach- ers’ with negative or poor learning experiences will have a lower level of TPACK. The factor analysis extracted 2 components underlying this construct, each component has the eigenvalue over 1 which account for the 71.20% of the variance. This means that there are two constructs underlying the 6 items in TLE domain. Two items (TUTTEE53 ‘When technologies are used in my classroom, it is the lec- turers who use technologies most of the time’ and TUTTEE54 ‘I am allowed to use any technology software/hardware I am familiar with in the class- rooms’) needed special attention since their factor lo adin gs were. 267 and .003 res pect ivel y. TUTTEE53 item was then deleted since the ques- tion might be redundant with TUTTEE49 (‘My lecturers use technologies in the classrooms’) and the information asked was in fact implied in TUTTEE52 (‘When technologies are used in my classroom, it is the students who use technologies most of the time’). Item TUTTEE54 was elimi- nated by considering its irrelevancy with the con- struct. The internal consistency of the set of items un- der each domain was investigated using Cronbach’s alpha technique. Table 2 illustrates the internal consistency from each domain. DOMAIN NAME CRONBACH ALPHA Technological Knowledge (TK) .82 Technological Content Knowledge (TCK) .86 Technological Pedagogical Knowledge (TPK) .82 Technological Pedagogical Content Knowledge (TPCK) .87 Technology-related Learning Experience (TLE) .67 TABLE 2: CRONBACH ALPHA FOR EACH DOMAIN The result in Table 1 indicates that the internal consistency reliability for Technology-related Learn- ing Experience was low while the other domains had satisfactory scale. The questionable items 23 Journal of Foreign Language Teaching & Learning Vol.2 No.2 July 2017 within the Technology-related Learning Experience domain were examined. In line with the result of the factor analysis for this construct, items TUTTEE53 and TUTTEE54 needed to be dropped to increase the reliability of this domain. The Cronbach’s alpha increased to.78 when these two items were dropped. As a result, a total of 21 items were eliminated from the survey, including 19 TK items and 2 TLE items. CONCLUSION Efforts toward building the validity and reliabil- ity of the instrument had been performed by this study. The results suggest that this instrument is considered acceptable to be used in a larger scale of work that aims to investigate the role of pre- service teachers’ experiences in learning to use ICT in their pedagogical practice in inf luencing the development of their TPACK. However, much work needs to be done with regards to further vali- dating and revising the instrument. Stronger va- lidity and reliability should be the focus of future studies. This can be done by conducting expert re- view to build content validity, applying rigorous sampling techniques, and conducting validity and reliability tests on the qualitative types of the items in this instrument. A valid and reliable instrument will be beneficial in providing accurate feedback on ICT-related teacher professional learning pro- grams. REFERENCES Angeli, C., & Valanides, N. (2009). Epistemologi- cal and met hodo logi cal issues f or t he conceptualization, development, and assessment of ICT–TPCK: Advances in technological peda- gogical content knowledge (TPCK). Computer & Education, 52, 154-168. Beckett, E. C., Wetzel, K., Chishlom, I. M., Zambo, R., Buss, R., Padgett, H., et al. (2003). Support- ing technology integration in K-8 multicultural classrooms through professional development. TechTrends, 47(5), 14-17. Belawati, T. (2005). UNESCO Meta-survey on the use of technologies in education: Indonesia ICT Use in Education. Retrieved from http:// www.unescobkk.org/ fileadmin/user_upload/ ict/Metasurvey/indonesia.pdf Borko, H. (2004). Professional development and teacher learning: Mapping the terrain. Educa- tional Researcher, 33(3). Campbell, A., McNamara, O., & Gilroy, P. (2004). Practitioner Research and Professional Devel- opment in Education. London: Paul Chapman. Cole, K., Simkins, M., & Penuel, W. R. (2002). Learning to Teach with Technology: Strategies VARIABLE NAME ITEM LABEL COMPONENT 1 2 3 4 5 6 7 TUTTEA3 Ability in operating Instant Message .291 .255 -.218 .619 -.406 -.252 -.337 TUTTEA5 Ability in operating Weblog .310 .628 -.332 .058 -.400 -.352 -.057 TUTTEA6 Ability in operating Electronic mailing list .081 .219 .191 .676 .245 .008 .513 TUTTEA7 Ability in operating Podcast .341 .447 .638 .163 -.137 -.134 .074 TUTTEA9 Ability in operating scanner .037 .185 .838 .193 -.050 .131 .055 TUTTEA12 Ability in operating Tablet computer .026 .536 .602 -.178 .038 -.316 -.214 TABLE 1: COMPONENT MATRIX OF PROBLEMATIC ITEMS UNDER THE TK DOMAIN http://www.unescobkk.org/ 24 Journal of Foreign Language Teaching & Learning Vol.2 No.2 July 2017 for Inservice Professional Development. Jour- nal of Technology and Teacher Education, 10(3), 431-455. Cook, K. F., Cella, D., Boespf lug, E. L., & Amtmann, D. (2010). Is less more? A prelimi- nary investigation of the number of response categories in self-reported pain. Retrieved from h t t p : / / s ch o l a r. g o o g l e u s e rc o n t e n t . c o m/ scholar?q=cache:X7rUpgwtRCgJ:scholar.google.com/ +Is+less+more%3F+A+preliminary+investigation+of+ the+number+of+response+categories+in+self- reported+pain&hl=en&as_sdt=0,5 Creswell, J. W. (2011). Educational research: Plan- ning, conducting, and evaluating quantitative and qualitative research (4th ed.). Boston: Pearson Education. Denning, T., & Selinger, M. (1999). Patterns of Change and Innovations in Pre-service Educa- tion. Paper presented at the Annual Conference of SITE ’99 (Society for Information Technol- ogy & Teacher Education, San Antonio, Texas. Desimone, L. (2009). Improving Impact Studies of Teachers’ Professional Development: Toward Better Conceptualizations and Measures. Edu- cational Researcher, 38(3), 181-199. Desimone, L. M., & Le Floch, K. C. (2004). Are we asking the right questions? Using cognitive interviews to improve surveys in education re- search. Educational Evaluation and Policy Analy- sis, 26(1), 1-22. Eraut, M. (2007). Learning from other people in the workplace. Oxford Review of Education, 33(4), 403-422. Field, A. (2009). Discovering statistics using SPSS (3rd ed.). London: SAGE. Finn, A., & Peng, L. (2009). How Many Response Categories Best Scale Stimuli? Retrieved from http://www.duplication.net.au/ANZMAC09/ papers/ANZMAC2009-146.pdf Garet, M. S., Porter, A. C., Desimone, L., Birman, B. F., & Yoon, K. S. (2001). What Makes Pro- fessional Development Effective? Results From a National Sample of Teachers. American Edu- cational Research Journal, 38(4), 915-945. Guskey, T. R. (2000). Evaluating professional de- velopment. Thousand Oaks, CA: Corwin Press. Hawley, W. D., & Valli, L. (1999). The essentials of professional development: a new consensus. In L. Darling-Hammond & G. Sykes (Eds.), Teaching as the learning profession: Handbook of theory and practice. San Francisco, CA: Jossey-Bass. Kariuki, M., Franklin, T., & Duran, M. (2001). A technology partnership: Lessons learned by mentors. Journal of Technology and Teacher Education, 9(3), 407-417. Keller, J. B., Hixon, E., Bonk, C. J., & Ehman, L. H. (2004). Professional development that in- creases technology integration by K-12 teachers: Influence of the TICKIT program. Retrieved from http://www.itdl.org/journal/mar_08/ article01.htm Koehler, M. J., & Mishra, P. (2005). Teachers Learn- ing Technology by Design. Journal of Comput- ing in Teacher Education, 21(3), 94-102. Koehler, M. J., Mishra, P., & Yahya, K. (2007). Trac- ing the development of teacher knowledge in a design seminar: Integrating content, pedagogy, and technology. Computers & Education, 49, 740-762. Koehler, M. J., & Mishra, P. (2008). Introducing TPCK. In AACTE (Ed.), The handbook of tech- nological pedagogical content knowledge (TPCK) for educators. New York: Routledge. http://www.duplication.net.au/ANZMAC09/ http://www.itdl.org/journal/mar_08/ 25 Journal of Foreign Language Teaching & Learning Vol.2 No.2 July 2017 Koh, J. H. L., Chai, C. S., & Tsai, C. C. (2010). Examining the technological pedagogical con- tent knowledge of Singapore pre-service teach- ers with a large-scale survey. Journal of Com- puter Assisted Learning, 26(6), 563-573. Lawless, K. A., & Pellegrino, J. W. (2007). Profes- sional development in integrating technology into teaching and learning: Knowns, unknowns, and ways to pursue better questions and answers. Review of Educational Research, 77(4), 575-614. Lietz, P. (2008). Questionnaire design in attitude and opinion research: Current state of an art. Priorisierung in der Medizin, FOR 655(13). Retrieved from http://www.priorisierung-in- d e r - m e d i z i n . d e / d o c u m e n t s / F O R 6 5 5 _ Nr13_Lietz.pdf. Ministry of National Education. (2007a). Peraturan menteri pendidikan nasional republik Indone- sia nomor 16 tahun 2007 tentang standar kualifikasi akademik dan kompetensi guru [The minister of national education regulation no. 16/2007 about teacher academic qualification and competency standard]. Jakarta. Ministry of National Education. (2007b). Peraturan menteri pendidikan nasional republic Indone- sia nomor 41 tahun 2007 tentang standard proses untuk satuan pendidikan dasar dan menengah [The minister of national education regulation no. 41/2007 about the standardize processed for primary and secondary level of education]. Jakarta. Ministry of National Education. (2009). Peraturan menteri pendidikan nasional republik Indone- sia nomor 78 tahun 2009 tentang penyelengga- raan sekolah bertaraf internasional pada jenjang pendidikan dasar dan menengah [The minister of national education regulation no. 78/2009 about the implementation of interna- tional standard school at primary and second- ary level of education]. Jakarta. Ministry of National Education. (2010). Rencana strategis pendidikan 2010-2014 [Strategic plan in education 2010-2014]. Jakarta. Mishra, P., & Koehler, M. J. (2006). Technological pedagogical content knowledge: A framework for teacher knowledge. Teachers College Record, 108(6), 1017-1054. Mulqueen, W. E. (2001). Technology in the Class- room: Lessons learned through professional development. Education, 122(2), 248-256. Nagata, C., Ido, M., Shimizu, H., Misao, A., & Matsuura, H. (1996). Choice of response scale for health measurement: Comparison of 4, 5, and 7-point scales and visual analogue scale. Jour- nal of Epidemiology, 6(4), 192-197. Podsakoff, P. M., MacKenzie, S. B., & Lee, J. (2003). Common Method Biases in Behavioral Re- search: A Critical Review of the Putnam, R. T., & Borko, H. (2000). What Do New Views of Knowledge and Thinking Have to Say About Research on Teacher Learning?. Educa- tional Researcher, 29(1), 4-15. Sahin, I. (2011). Development of survey of techno- logical pedagogical and content knowledge (TPACK). The Turkish Online Journal of Edu- cational Technology, 10(1), 97-105. Salant, P. A., & Dillman, D. A. (1994). How to conduct your own survey. New York: Wiley & Sons, Inc. Schmidt, D. A., Baran, E., Thompson, A. D., Mishra, P., Koehler, M. J., & Shin, T. S. (2009/ 2010). Technological pedagogical content knowl- edge (TPACK): The development and validation of an assessment instruments for preservice http://www.priorisierung-in- 26 Journal of Foreign Language Teaching & Learning Vol.2 No.2 July 2017 teachers. Journal of Research on Technology in Education, 42(2), 123-149. Shulman, L. S. (1986). Those who understand: Knowledge growth in teaching. Educational Researcher, 15(2), 4-14. Shulman, L. S. (1987). Knowledge and teaching: Foundations of the new reform. Harvard Edu- cational Review, 57, 1-22. UNESCO (2007). Initiating and managing Sch oolN ets. Ret riev ed f rom http :// www2.unescobkk.org/elib/publications/111/ Van Olphen, M. (2008). TPCK an integrated framework for educating world language teach- ers. In AACTE (Ed.), The handbook of techno- logical pedagogical content knowledge (TPCK) for educators. New York: Routledge. http://www2.unescobkk.org/elib/publications/111/